Data Pump Export workers are waiting on ‘wait for unread message on broadcast channel’

Version:

Oracle 11.2.0.3.0 - Production

Issue:

A database full export job triggered couple of days back is very slow or hung. Around 2 TB of dump files were generated, but it seemed to be hung after that.

The job status is shown as "Executing" but the dump file size showed no progress and stayed at same value for more than a day.

Analysis:

There were no messages or warnings in the alert log file.
Export status showed as executing and there was n't any errors.


Export> status

Job: SYS_EXPORT_FULL_01

  Operation: EXPORT

  Mode: FULL

  State: EXECUTING

  Bytes Processed: 2,104,376,791,712

  Percent Done: 94

  Current Parallelism: 10

  Job Error Count: 0

  Dump File: /backup/prod/DB/orcl/expdp/orcl_datapump_%u.dmp  

An event 10046 trace showed that it is waiting on the following

"wait for unread message on broadcast channel"

There is a support Doc for similar issue. Reason is the statistics are not up-to-date

Solution:

1) Gather statistics on the data dictionary and system tables.

EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

EXEC DBMS_STATS.GATHER_SYSTEM_STATS;

EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS (null);

2) Restart the EXPDP job 

After gathering statistics. The performance was back to normal.

Restarting explorer.exe process in Windows 2008 server connected remotely

Version: 

Version: Windows Server 2008; Release 2 64 Bit

Issue:

Accidentally killed explorer.exe process via Process Explorer tool in Windows 2008 server. The connection to the server was via Remote Desktop

Killing the explorer resulted in all windows getting closed. No start screen or taskbar either

Solution:

1. Press  <CTRL> <ALT> <END>  keys on the remote desktop session. This command is similar to   <CTRL> <ALT>  <DEL>  on local system , which will open the Task Manager

2. In the task manager window click "New Task" under Applications tab.

3. In the New Task window opened, type "explorer.exe" and  click OK. 

This will open a new instance of explorer

FRM-92101: There was a failure in the Forms Server during startup

Version:

Oracle R12 12.1.3 in OEL 5 64 bit

Issue: 

Forms is not accessible, on accessing forms from URL it threw error. 
All the services were started up using adstrtal.sh script and there was no failures detected at the script startup time

Error:

The following error was reported on accessing forms URL.

FRM-92101: There was a failure in the Forms Server during startup. This could happen due to invalid configuration. 

Analysis:

Checked the forms log files available under $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1. From the application.log, the error messages were identified

15/02/02 23:52:34.437 formsweb: 10.1.3.4.0 Started
15/02/02 23:57:03.843 formsweb: ListenerServlet init()
15/02/02 23:57:05.159 formsweb: Forms session <1> aborted: runtime process failed during startup with errors /u01/E-BIZ/apps/tech_st/10.1.2/bin/frmweb: error while loading shared libraries: libXm.so.2: cannot open shared object file: No such file or directory

15/02/02 23:57:05.159 formsweb: Forms session <1> exception stack trace:
oracle.forms.engine.RunformException: Forms session <1> failed during startup: no response from runtime process
        at oracle.forms.servlet.RunformProcess.connect(Unknown Source)

Went through the following Oracle support docs, and identified that the error is due to missing RPM
The RPM which was missing is openmotif21

After EBS Fresh Install accessing any Forms Responsibility fails with error "FRM-92101: Forms Server" (Doc ID 1192205.1)
Check RPMs from Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1) 

Solution:

1) Installed the missing RPM after downloading it form a site. 

In OEL 5, the missing RPM is openmotif21-2.1.30-11.EL5.i386.rpm. you can verify the document ID (402310.1 for 32 bit and 416305.1 for 64 bit) for correct RPM required for your platform. 

The RPM, openmotif21-2.1.30-11.EL5.i386.rpm is not part of the OS distribution media and must be downloaded separately

[root@Appsnode tmp]rpm -ivh openmotif21-2.1.30-11.EL5.i386.rpm
Preparing...                ########################################### [100%]
  1:openmotif21           ########################################### [100%]
[root@Appsnode  tmp]# rpm -qa|grep openmotif21

openmotif21-2.1.30-11.EL5

Note:
For OEL 6, the rpm required is openmotif21-2.1.30-11.EL6.i686.rpm

2) Restart the middletier after applying the package.

Can't load 'Sys/Hostname/Hostname.so' for module Sys::Hostname: wrong ELF class: ELFCLASS64

Version:

Oracle R12 version 12.1.3 in OEL 64 bit

Issue:

Error messages were found in the adformsctl.sh start.

adformsctl.sh: check the logfile /u01/E-BIZ/inst/apps/dmw24rep_slce20cn14/logs/appl/admin/log/adformsctl.txt for more information ...  

.end std out.
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/E-BIZ/inst/apps/dmw24rep_slce20cn14/logs/appl/rgf/TXK/txkChkFormsDeployment_Mon_Feb_2_20_31_21_2015/txkChkFormsDeployment_Mon_Feb_2_20_31_21_2015.log 



Error:

On checking the logfile, it is understood that the issue is with PERL Paths in the context file.

Can't load '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so' for module Sys::Hostname: /usr/lib64/perl5/5.8.8/x86_64-linux-thread
-multi/auto/Sys/Hostname/Hostname.so: wrong ELF class: ELFCLASS64 at /usr/lib64/
perl5/5.8.8/x86_64-linux-thread-multi/XSLoader.pm line 70.
 at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Sys/Hostname.pm line 23
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/E-BIZ/inst/apps//r12env_host2/logs/appl
/rgf/TXK/txkChkFormsDeployment_Mon_Feb_2_19_58_53_2015/txkChkFormsDeployment_Mon
_Feb_2_19_58_53_2015.log 

Reason:

The environment is created as shared storage multi node. The second node was cloned from the first, but in the context file, the PERL related paths were not identical to that in first node

The paths in failed node

<ADPERLPRG oa_var="s_adperlprg" osd="unix">/usr/bin/perl</ADPERLPRG>

<PERL5LIB oa_var="s_perl5lib" osd="LINUX_X86-64">/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/site_perl/5.8.8:/usr/lib/perl5/site_perl:/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/vendor_perl/5.8.8:/usr/lib/perl5/vendor_perl:/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/5.8.8:.:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl</PERL5LIB>

Fix:

Edited the perl paths (ADPERLPRG  and PERL5LIB) in the context file and made it identical to that in first node and restarted the services.

Paths after editing:

<ADPERLPRG oa_var="s_adperlprg" osd="unix">/u01/E-BIZ/apps/tech_st/10.1.3/perl/bin/perl</ADPERLPRG>

<PERL5LIB oa_var="s_perl5lib" osd="LINUX_X86-64">/u01/E-BIZ/apps/tech_st/10.1.3/perl/lib/5.8.3:/u01/E-BIZ/apps/tech_st/10.1.3/perl/lib/site_perl/5.8.3:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl:/u01/E-BIZ/apps/tech_st/10.1.3/Apache/Apache/mod_perl/lib/site_perl/5.8.3/i686-linux-thread-multi</PERL5LIB>

Restart service in second node after this.