DB startup failure with ORA-27154, ORA-27300, ORA-27301 and ORA-27302

Version:

Oracle 12.1.0.1.0 64 bit on OEL 6.3

Issue:

Unable to start up 12c DB

Error:

SQL> startup;
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 17
ORA-27301: OS failure message: File exists
ORA-27302: failure occurred at: sskgpcreates

Troubleshoot:

Issue the command "sysresv" to view the currently allocated IPC resources

bash-4.1$ sysresv

IPC Resources for ORACLE_SID "orcl" :
Shared Memory
ID              KEY
No shared memory segments used
Semaphores:
ID              KEY
3964951         0xfbf59bfc
3997720         0xfbf59bfd
4030489         0xfbf59bfe
4063258         0xfbf59bff
4096027         0xfbf59c00
Oracle Instance not alive for sid "orcl"

We have to identify and cleanup any stray shared memory segments and semaphores still owned by the previous startup of Oracle. 

To get a list of shared memory and semaphore id's:

ipcs -a

To delete shared memory:

ipcrm -m <shared memory id>

To delete semaphores:

ipcrm -s <semaphore id>

Issue above commands as root user.

Commands:

[root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x1c8f7e64 229377     jtavares   640        14680064   0                       
0x00000000 262146     jtavares   640        1035993088 0                       
0x5d0c776c 950275     jtavares   640        4096       0                       
0x00000000 983044     jtavares   640        4096       0                       
0xa6728ac4 948862985  jtavares   640        4096       0                       
0x00000000 948895754  jtavares   640        4096       0                       
0x00000000 1111818251 jtavares   640        4096       0                       
0x00000000 1111851020 jtavares   640        4096       0                       
0x00000000 1111883789 jtavares   640        4096       0                       
0x49cc319c 1111916558 jtavares   640        12288      0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x2155cb64 163842     jtavares   640        151       
0x2155cb65 196611     jtavares   640        151       
0x2155cb66 229380     jtavares   640        151       
0x2155cb67 262149     jtavares   640        151       
0x2155cb68 294918     jtavares   640        151       
0x39fd8dd4 425991     jtavares   640        152       
0x39fd8dd5 458760     jtavares   640        152       

[root@devdbhost1 ~]# ipcrm -m 229377
[root@devdbhost1 ~]# ipcrm -m 262146
[root@devdbhost1 ~]# ipcrm -m 950275
[root@devdbhost1 ~]# ipcrm -m 983044
[root@devdbhost1 ~]# ipcrm -m 948862985
[root@devdbhost1 ~]# ipcrm -m 948895754
[root@devdbhost1 ~]# ipcrm -m 1111818251
[root@devdbhost1 ~]# ipcrm -m 1111851020
[root@devdbhost1 ~]# ipcrm -m 1111883789
[root@devdbhost1 ~]# ipcrm -m 1111916558

[root@devdbhost1 ~]# ipcrm -s 163842
[root@devdbhost1 ~]# ipcrm -s 196611
[root@devdbhost1 ~]# ipcrm -s 229380
[root@devdbhost1 ~]# ipcrm -s 262149
[root@devdbhost1 ~]# ipcrm -s 294918
[root@devdbhost1 ~]# ipcrm -s 425991
[root@devdbhost1 ~]# ipcrm -s 458760
[root@devdbhost1 ~]# ipcrm -s 491529
[root@devdbhost1 ~]# ipcrm -s 1081362
[root@devdbhost1 ~]# ipcrm -s 1114131
[root@devdbhost1 ~]# ipcrm -s 1146900
[root@devdbhost1 ~]# ipcrm -s 1179669
[root@devdbhost1 ~]# ipcrm -s 1212438
[root@devdbhost1 ~]# ipcrm -s 3964951
[root@devdbhost1 ~]# ipcrm -s 3997720
[root@devdbhost1 ~]# ipcrm -s 4030489
[root@devdbhost1 ~]# ipcrm -s 4063258
[root@devdbhost1 ~]# ipcrm -s 4096027
[root@devdbhost1 ~]# ipcrm -s 3506204
[root@devdbhost1 ~]# ipcrm -s 3538973
[root@devdbhost1 ~]# ipcrm -s 3571742



root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages 

Restarted DB successfully after this

bsu.sh: java.lang.OutOfMemoryError: GC overhead limit exceeded

Version:

Oracle Weblogic server 10.3.6 in OEL 64 bit

Issue:

July 2015 PSU (WLS PATCH SET UPDATE 10.3.6.0.12) on weblogic server is failing with "OutOfMemoryError". Patching is failing immediately after it is invoked.

Error:

[oracle@soahost1 bsu]$ ./bsu.sh -install -patch_download_dir=/home/oracle/middleware/utils/bsu/cache_dir -patchlist=EJUW -prod_dir=/home/oracle/middleware/wlserver_10.3

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:448)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:464)
        at com.bea.plateng.patch.dao.cat.PatchCatalog.getPatchDependencies(PatchCatalog.java:56)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getInvalidatedPatchMap(PatchCatalogHelper.java:1621)
        at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:436)
        at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
        at com.bea.plateng.patch.PatchSystem.setCacheDir(PatchSystem.java:201)
        at com.bea.plateng.patch.Patch.main(Patch.java:281)

Reason:

The default java memory arguments  provided in bsu.sh file are not sufficient

Solution:

Increase the Java memory values in bsu.sh file (under $MW_HOME/utils/bsu) by adjusting the value MEM_ARGS values.

By default the values will be

MEM_ARGS="-Xms256m -Xmx512m"

I increased it to 1024-1024 as follows

"-Xms1024m -Xmx1024m"

Save the bsu.sh file and re-execute the patching command.

TNS-12555: TNS:permission denied with TNS-00525

Version:

Oracle EE 12.1.0.1.0 in Linux 64 bit

Issue:

Listener is not coming up when it is tried to start. We always ends up getting the same error log.

Errors:

TNSLSNR for Linux: Version 12.1.0.1.0 - Production
System parameter file is /home/oracle/product/121010/network/admin/listener.ora
Log messages written to /home/oracle/diag/tnslsnr/hostdb1/listener121010/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hostdb1.idc.oracle.com)(PORT=1211)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1211)))
TNS-12555: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00525: Insufficient privilege for operation
   Linux Error: 1: Operation not permitted

Listener failed to start. See the error message(s) above...

Reason: 

In most cases this will happen when "root" user was used for previous start of listener.That is the reason why we are getting permission denied and Insufficient privilege in error stack.

Solution:

1. cd /var/tmp

2. Check the ownership of the cd /var/tmp/.oracle folder. It should be such a way that the owner is the <oracle> owner and the group '<dba>' group of the oracle user.

In this case .oracle was owned by root as shown below

drwxrwxrwt   2 root root  4096 Apr 28 16:59 .oracle


3. Change the user and group using the appropriate command:

I have used the following commands as root user

chown oracle .oracle
chgrp dba .oracle
chmod 777 .oracle

After the change the folder permissions are as follows

drwxrwxrwt   2 oemora dba   4096 Apr 28 17:04 .oracle

4. Start the TNS Listener.

It should start normally now.

ld.so.1: oracle: fatal: libclntshcore.so.12.1: open failed: No such file or directory

Version:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

OS :Oracle Solaris on SPARC

Issue:

After applying PSU 12.1.0.2.3, connection to DB to do the startup, via sqlplus was failing. Was not able to connect to DB via sqlplus. The environment variables were set via ".oraenv" command.

Error:

sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Fri Jun 12 07:15:23 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ld.so.1: oracle: fatal: libclntshcore.so.12.1: open failed: No such file or directory
ERROR:
ORA-12547: TNS:lost contact

Verification:

The file mentioned () was present under the $ORACLE_HOME/lib folder.

 ls -ltr $ORACLE_HOME/lib | grep libclntshcore

-rwxrwxr-x   1 oradba   oinstall 7275016 Jun 12 07:10 libclntshcore.so.12.1

lrwxrwxrwx   1 oradba   oinstall      21 Jun 12 07:10 libclntshcore.so -> libclntshcore.so.12.1

So it could be due to the location missing in the Library path variable.

Solution:

Set the environment variable, LD_LIBRARY_PATH to $ORACLE_HOME/lib.

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

After this able to connect to sqlplus without any issues.

Oracle Weblogic: Setting independant JVM values for Managed Server in setDomainEnv.sh

Version: 

WLS 10.3.6 or FMW Infrastructure 12.1.3

Requirement:

Set the managed server to start with 1GB heap size and 1GB permgen space.

Steps:

1. Open the setDomainEnv.sh (under $DOMAIN_HOME/bin) in a text editor like vi

2. Search the file for the line starting with "IF USER_MEM_ARGS"

    Search for the commented out line starting with "IF USER_MEM_ARGS the environment variable     is set" in the file

3. Add the memory settings for your managed server below that line.
   
    Add the required settings for your managed server. 

Eg: 

# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values

if [ "${SERVER_NAME}" == "PRD_ManServer1" ] ; then
     USER_MEM_ARGS="-Xms1024m -Xmx1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m"
export USER_MEM_ARGS
fi
f [ "${USER_MEM_ARGS}" != "" ] ; then
        MEM_ARGS="${USER_MEM_ARGS}"
        export MEM_ARGS
fi

where PRD_MasServer1 is the name of my managed server. I have set both Java heap and permgen space to 1024 (both low and high values) in the above example.

4. Save the file and quit. Restart the managed service

5. Verify the change

    Grep weblogic process to make sure the change is effective

Eg: 

ps -ef|grep weblogic

/java/bin/java -server -Xms1024m -Xmx1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m -Dweblogic.Name=PRD_ManServer1

ODI-1228 with ORA-25153: Temporary Tablespace is Empty

Version : 

Oracle DB 11.2.0.4.0 - 64bit Production

Issue:

ETL using ODI failing with following error
   
ODI-1228: Task INT_HDI_ARC_HIERARCHY (Integration) fails on the target ORACLE connection HDI.
Caused By: java.sql.SQLException: ORA-25153: Temporary Tablespace is Empty

Analysis:

The error lookup with oerr command showed this

$ oerr ora 25153
25153, 00000, "Temporary Tablespace is Empty"
// *Cause: An attempt was made to use space in a temporary tablespace with no files.
// *Action: Add files to the tablespace using ADD TEMPFILE command.

Verified that the default tablespace of both DB and the DB user doing the ETL is temp1

SQL>  select property_value from database_properties where property_name like '%DEFAULT_TEMP_TABLESPACE%';

PROPERTY_VALUE
--------------------------------------------------------------------------------
TEMP1

SQL> select TEMPORARY_TABLESPACE from dba_users where USERNAME='HDI_UPGR2';

TEMPORARY_TABLESPACE
------------------------------
TEMP1

But when we select the tablepace name based on dba_temp_files view, TEMP1 is missing and onlt TEMP tablespace is seen as shown below.

SQL> select tablespace_name from dba_temp_files;

TABLESPACE_NAME
------------------------------
TEMP

Reason:

The file is deleted at OS level.

We cannot recreate the TEMP1 tablespace because the entry is still available in data dictionary and only the temp file associated with the tablespace is physically removed. If we try to recreate we will get following error.

ORA-01543: tablespace 'TEMP1' already exists

Solution:

Add a file to the TEMP1 tablespace using ALTER TABLESPACE command as shown here

SQL> alter tablespace temp1 add tempfile '<FILE_PATH>/temp01.dbf' size 100m autoextend on next 1m maxsize unlimited;

Tablespace altered.

Now when we query dba_temp_files, we can get TEMP1 as well.


SQL> select tablespace_name from dba_temp_files;

TABLESPACE_NAME
------------------------------
TEMP
TEMP1

ETL succeded after that

Adding additional managed server in new machine to the existing weblogic cluster

Version: 

WLS 10.3.6 - Clustered environment

Requirement:

Add a 3rd managed server to the existing weblogic cluster setup. Currently it has 1 Admin server and 1 Managed server running on Node 1 and another managed server running on Node 2. Our plan is to add the 3rd Managed server in Node 3 to the cluster.

Note:

Shared Storage: Storage is shared across all the nodes and same OS user/group configuration is used.  If the storage is not shared, then we need to pack the domain to new server and unpack it there.

We can add Managed Servers to a domain/cluster to increase the capacity of the system.  When a Managed Server is added to a cluster, it inherits the applications and services that are targeted to the cluster. So if we have deployed an application on top of the cluster, same would be reflected immediately on the managed server, which was added newly, when it is started up.

Steps 1: Add Managed Server

1. Login to Weblogic admin console URL
2. Click on "Lock & Edit" at the top Left corner of the console URL
3. In the left pane, expand Environment, then select Servers.
4. In the Servers table, click New button - It opens a new page -> Create a New Server - Domain Name
5. Enter the following information in that page
    - Server Name: Give the new Managed Server name (Eg: ManagedServer2)
    - Server Listen Address: Give fully qualified server name with domain here           (eg: myserver.mydomain)
    - Server Listen Port: Give same port number as of the existing managed servers since this is going to run in a different server. If same server, then port number should be different
    - Choose - "Yes, make this server a member of an existing cluster" , if this server is to be part of an existing cluster
6. Click Next
7. Review the details we provided and click Finish

Managed Server is now created. Next step is to create a machine and associate that with this managed server

Steps 2: Create a new Machine

1. Login to Weblogic admin console URL
2. Click on "Lock & Edit" at the top Left corner of the console URL
3. In the left pane, expand Environment, then select "Machines"
4. Click New button in the Machines table. It opens a new page -> Create a New Machine - Domain Name
5. Enter the following information in that page
    - Name: A suitable name for the machine. I normally give the hostname of the server without the domain name here
    - Machine OS: Choose Unix if it is a Unix flavour (Other options are virtual and other)
6. Click Next -> This will open the "Node Manager Properties" page. Enter the following information in that page
    - Type: SSL (protocol to communicate with the Nodemanager)
    - Listen Address: Give fully qualified server name with domain
    - Listen Port: Listen port. 5556 is the default
7. Click Finish

Now add managed server to this machine

Steps 3: Assign Managed Server to the Machine

1. Click on the machine name we just created. - It will open the Settings for <the new Machine> page.
2. Click on the "Servers" tab in that page
3. In the Servers table, click Add
4. Choose "Select an existing server, and associate it with this machine" option and choose the managed server we created above
5. Click Finish

Finally click on "Release Configuration"

Start up the nodemanager directly from the server where it is configured to run.
Verify that the machine status is "Reachable" in Environment => Machines => <Machine Name> => Monitoring tab
Go back to servers => Click control and select the managed server we created above. Press Start

Now we have the managed server running as part of the cluster.

The Java RunTime Environment was not found at /tmp/OraInstall/jdk/jre/bin/java

Version:

Oracle Fusion Middleware Forms and Reports version 11.1.2.1.0 on Linux 64 bit

Issue:

Run installer errors out with java issue

Error:

bash-4.1$ ./runInstaller 
Starting Oracle Universal Installer...

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-03-24_12-33-40AM. Please wait ...The Java RunTime Environment was not found at /tmp/OraInstall2015-03-24_12-33-40AM/jdk/jre/bin/java. Hence, the Oracle Universal Installer cannot be run.
Please visit http://www.javasoft.com and install JRE version 1.3.1 or higher and try again.
: No such file or directory

Analysis:

Java 1.6 update 65 is already installed and JAVA_HOME environment variable is set as well as PATH variable has $JAVA_HOME/bin entry. Verified the settings with which java and java -version commands


Solution:

oraparam.ini file under the software staged directory path /software/FR11121/Disk1/install/linux64 had an entry JRE_SCRATCH_LOC

It was set to 
JRE_SCRATCH_LOC=jdk/jre

Commented out above line using VI editor and saved the file

vi oraparam.ini

#JRE_SCRATCH_LOC=jdk/jre

Reinvoked runInstaller.

This time runInstaller prompted for java path and I gave the already installed path and installer took it


bash-4.1$ ./runInstaller 
Starting Oracle Universal Installer...


Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-03-23_08-06-23AM. Please wait ...
Please specify JRE/JDK location ( Ex. /home/jre ), <location>/bin/java should exist :/install/product/java

[VirtualHost: main] (13)Permission denied: make_sock: could not bind to address 0.0.0.0:80

Version:

Oracle HTTP Server 11.1.1.7.0 in Linux 64 Bit

Issue:

opmnctl startall command errors out when invoked with the OS user who installed the product.

Error:

From $INSTANCE_HOME/diagnostics/logs/OHS/ohs1/ohs1.log the following errors are seen

[2015-03-03T11:08:11.5520+00:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: webtier1] [host_addr: *.*.*.*] [pid: 2414] [tid: 139750680717120] [user: oracle] [VirtualHost: main]  caught SIGTERM, shutting down

[2015-03-03T16:38:37.7081+05:30] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: webtier1] [host_addr: *.*.*.*] [pid: 5776] [tid: 140408010676032] [user: oracle] [VirtualHost: main] (13)Permission denied:  make_sock: could not bind to address 0.0.0.0:80

[2015-03-03T16:38:37.7081+05:30] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [core.c] [host_id: webtier1] [host_addr: *.*.*.*] [pid: 5776] [tid: 140408010676032] [user: oracle] [VirtualHost: main]  no listening sockets available, shutting down

Reason:

Seeing the logs, we identified the main reason is Permission denied to use the port 80
Permission denied:  make_sock: could not bind to address 0.0.0.0:80

You will get the following error if you try to start Oracle HTTP Server on a port below 1024 as non-root user

Oracle HTTP Server will not start on ports below 1024 because root privileges are needed to bind these ports.

On a UNIX system the TCP/IP port numbers below 1024 are special in that only processes with root privileges are allowed to listen on those ports.

Solution:

By default, Oracle HTTP Server runs as a non-root user (the user that installed Oracle Fusion Middleware). Therefore, on UNIX systems, if you plan on running Oracle HTTP Server on a privileged port (for example, port 80, 443 etc), you must enable Oracle HTTP Server to run as root user

1. Take a backup of  ".apachectl file" (under $WT_HOME/ohs/bin directory) as the software owner (in my case, oracle) to keep current permissions on the file.

Please note that it is a hidden file (.apachectl). There is an unhidden file apachectl also in same dir.

2. Connect as root user and change to the above directory

[root@webtier1 ~]# cd $MW_HOME/Oracle_WT1/ohs/bin

3. Change permissions of the file using below given 2 commands

chown root .apachectl

chmod 6750 .apachectl

[root@webtier1 bin]# chown root .apachectl

[root@webtier1 bin]# chmod 6750 .apachectl

Now the permission of the .apachectl file would be as follows

-rwsr-s---  1 root   dba  13278 Feb  8  2013 .apachectl

4. Exit root and connect back as the S/W owner user (oracle)

5. Edit httpd.conf file

cd $MW_HOME/Oracle_WT1/ohs/conf

Backup httpd.conf before editing

6. Search for the User/Group strings in the file. Then uncomment or add the installation user details.

For eg: originally in my httpd.conf the entries were

#User USER_TEMPLATE

#Group GROUP_TEMPLATE

</IfModule>

The above lines are modified to

User oracle

Group dba

</IfModule>

where oracle is the user with which I installed the Webtier and dba is the primary group of oracle

7. Start the opmnctl service as S/W owner(oracle)

Data Pump Export workers are waiting on ‘wait for unread message on broadcast channel’

Version:

Oracle 11.2.0.3.0 - Production

Issue:

A database full export job triggered couple of days back is very slow or hung. Around 2 TB of dump files were generated, but it seemed to be hung after that.

The job status is shown as "Executing" but the dump file size showed no progress and stayed at same value for more than a day.

Analysis:

There were no messages or warnings in the alert log file.
Export status showed as executing and there was n't any errors.


Export> status

Job: SYS_EXPORT_FULL_01

  Operation: EXPORT

  Mode: FULL

  State: EXECUTING

  Bytes Processed: 2,104,376,791,712

  Percent Done: 94

  Current Parallelism: 10

  Job Error Count: 0

  Dump File: /backup/prod/DB/orcl/expdp/orcl_datapump_%u.dmp  

An event 10046 trace showed that it is waiting on the following

"wait for unread message on broadcast channel"

There is a support Doc for similar issue. Reason is the statistics are not up-to-date

Solution:

1) Gather statistics on the data dictionary and system tables.

EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

EXEC DBMS_STATS.GATHER_SYSTEM_STATS;

EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS (null);

2) Restart the EXPDP job 

After gathering statistics. The performance was back to normal.

Restarting explorer.exe process in Windows 2008 server connected remotely

Version: 

Version: Windows Server 2008; Release 2 64 Bit

Issue:

Accidentally killed explorer.exe process via Process Explorer tool in Windows 2008 server. The connection to the server was via Remote Desktop

Killing the explorer resulted in all windows getting closed. No start screen or taskbar either

Solution:

1. Press  <CTRL> <ALT> <END>  keys on the remote desktop session. This command is similar to   <CTRL> <ALT>  <DEL>  on local system , which will open the Task Manager

2. In the task manager window click "New Task" under Applications tab.

3. In the New Task window opened, type "explorer.exe" and  click OK. 

This will open a new instance of explorer

FRM-92101: There was a failure in the Forms Server during startup

Version:

Oracle R12 12.1.3 in OEL 5 64 bit

Issue: 

Forms is not accessible, on accessing forms from URL it threw error. 
All the services were started up using adstrtal.sh script and there was no failures detected at the script startup time

Error:

The following error was reported on accessing forms URL.

FRM-92101: There was a failure in the Forms Server during startup. This could happen due to invalid configuration. 

Analysis:

Checked the forms log files available under $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1. From the application.log, the error messages were identified

15/02/02 23:52:34.437 formsweb: 10.1.3.4.0 Started
15/02/02 23:57:03.843 formsweb: ListenerServlet init()
15/02/02 23:57:05.159 formsweb: Forms session <1> aborted: runtime process failed during startup with errors /u01/E-BIZ/apps/tech_st/10.1.2/bin/frmweb: error while loading shared libraries: libXm.so.2: cannot open shared object file: No such file or directory

15/02/02 23:57:05.159 formsweb: Forms session <1> exception stack trace:
oracle.forms.engine.RunformException: Forms session <1> failed during startup: no response from runtime process
        at oracle.forms.servlet.RunformProcess.connect(Unknown Source)

Went through the following Oracle support docs, and identified that the error is due to missing RPM
The RPM which was missing is openmotif21

After EBS Fresh Install accessing any Forms Responsibility fails with error "FRM-92101: Forms Server" (Doc ID 1192205.1)
Check RPMs from Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1) 

Solution:

1) Installed the missing RPM after downloading it form a site. 

In OEL 5, the missing RPM is openmotif21-2.1.30-11.EL5.i386.rpm. you can verify the document ID (402310.1 for 32 bit and 416305.1 for 64 bit) for correct RPM required for your platform. 

The RPM, openmotif21-2.1.30-11.EL5.i386.rpm is not part of the OS distribution media and must be downloaded separately

[root@Appsnode tmp]rpm -ivh openmotif21-2.1.30-11.EL5.i386.rpm
Preparing...                ########################################### [100%]
  1:openmotif21           ########################################### [100%]
[root@Appsnode  tmp]# rpm -qa|grep openmotif21

openmotif21-2.1.30-11.EL5

Note:
For OEL 6, the rpm required is openmotif21-2.1.30-11.EL6.i686.rpm

2) Restart the middletier after applying the package.

Can't load 'Sys/Hostname/Hostname.so' for module Sys::Hostname: wrong ELF class: ELFCLASS64

Version:

Oracle R12 version 12.1.3 in OEL 64 bit

Issue:

Error messages were found in the adformsctl.sh start.

adformsctl.sh: check the logfile /u01/E-BIZ/inst/apps/dmw24rep_slce20cn14/logs/appl/admin/log/adformsctl.txt for more information ...  

.end std out.
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/E-BIZ/inst/apps/dmw24rep_slce20cn14/logs/appl/rgf/TXK/txkChkFormsDeployment_Mon_Feb_2_20_31_21_2015/txkChkFormsDeployment_Mon_Feb_2_20_31_21_2015.log 



Error:

On checking the logfile, it is understood that the issue is with PERL Paths in the context file.

Can't load '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so' for module Sys::Hostname: /usr/lib64/perl5/5.8.8/x86_64-linux-thread
-multi/auto/Sys/Hostname/Hostname.so: wrong ELF class: ELFCLASS64 at /usr/lib64/
perl5/5.8.8/x86_64-linux-thread-multi/XSLoader.pm line 70.
 at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Sys/Hostname.pm line 23
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/E-BIZ/inst/apps//r12env_host2/logs/appl
/rgf/TXK/txkChkFormsDeployment_Mon_Feb_2_19_58_53_2015/txkChkFormsDeployment_Mon
_Feb_2_19_58_53_2015.log 

Reason:

The environment is created as shared storage multi node. The second node was cloned from the first, but in the context file, the PERL related paths were not identical to that in first node

The paths in failed node

<ADPERLPRG oa_var="s_adperlprg" osd="unix">/usr/bin/perl</ADPERLPRG>

<PERL5LIB oa_var="s_perl5lib" osd="LINUX_X86-64">/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/site_perl/5.8.8:/usr/lib/perl5/site_perl:/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/vendor_perl/5.8.8:/usr/lib/perl5/vendor_perl:/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/5.8.8:.:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl</PERL5LIB>

Fix:

Edited the perl paths (ADPERLPRG  and PERL5LIB) in the context file and made it identical to that in first node and restarted the services.

Paths after editing:

<ADPERLPRG oa_var="s_adperlprg" osd="unix">/u01/E-BIZ/apps/tech_st/10.1.3/perl/bin/perl</ADPERLPRG>

<PERL5LIB oa_var="s_perl5lib" osd="LINUX_X86-64">/u01/E-BIZ/apps/tech_st/10.1.3/perl/lib/5.8.3:/u01/E-BIZ/apps/tech_st/10.1.3/perl/lib/site_perl/5.8.3:/u01/E-BIZ/apps/apps_st/appl/au/12.0.0/perl:/u01/E-BIZ/apps/tech_st/10.1.3/Apache/Apache/mod_perl/lib/site_perl/5.8.3/i686-linux-thread-multi</PERL5LIB>

Restart service in second node after this.

java.sql.SQLException: IO Error: Got minus one from a read call and ORA-27300

Version:

ODI 11.1.1.6.0 with Oracle DB 11.2.0.4.0 in Linux X86_64

Issue:

ODI and Informatica connections to the target  DB is failing with SQL exception. New connections are not getting created.

Error:

ODI-1217: Session LOAD_ENCOUNTER_TYPE_DIMENSION (2576017) fails with return code 8000. 
ODI-1226: Step ENCOUNTER_TYPE_CODE fails after 1 attempt(s). 
ODI-1228: Task ENCOUNTER_TYPE_CODE (Variable) fails on the target  connection ORACLE_HDM. 
Caused By: java.sql.SQLException: IO Error: Got minus one from a read call 

From the database alert log file, we got that the actual cause of the error could be

Process P097 died, see its trace file
Process startup failed, error stack:
Errors in file /export/oracle/diag/rdbms/orcl/orcl/trace/orcl_psp0_29459.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process P095 died, see its trace file

Analysis:

The error messages indicate that oracle has problem in forking required processes. This could be due to the limit set at the OS level per user.
There were 2 databases running on the same host and both had similar errors. The processes parameter were set at 2000 for each of them.

Following action items were executed as per Oracle support notes.

Increased nproc process limit to 16384 (/etc/security/limits.conf file)
Doubled the pid_max value to 72128 using command sysctl -w kernel.pid_max=72128
Restarted both the databases and listener

Still the problem persisted and no connections were possible.

Solution:

Manually edited the /etc/security/limits.d/90-nproc.conf file and updated the nproc value to 16384 from original value of 1024.
The entry in the above file explicitly overrides the settings in security.conf for the number of processes. It is a bug in some versions of Linux which enforces the old ulimit value of 1024 nproc's and overrides /etc/security/limits.conf custom settings.

The 90-nproc.conf file after editing will have entries as follows

*          soft    nproc     16384 
root       soft    nproc     unlimited

After editing its value to 16384 in 90-nproc.conf, the issue disappeared. It allowed the creation of more number of processes in the system

ORA-12520: TNS:listener could not find available handler for requested type of server

Version:

DB Version: 12.1.0.2.0

Issue: 

Connections to the DB are not happening either from sqlplus or from sqldeveloper. Tnsping command works fine

Error:

ORA-12520: TNS:listener could not find available handler for requested type of server. 

Reason:

When we log on to the DB host and connects directly without the tns service name, we get the actual error.

ORA-00020: maximum number of processes (300) exceeded

The error is due to Processes parameter value set in the database getting exceeded.

Fix:

Increase the parameter value in the database. We need to restart the DB to make the change effective since it is not dynamically modifiable.

SQL> alter system set processes=1000 scope=spfile;

System altered.

Shutdown and restart DB

java.io.FileNotFoundException: .\config\jps-config.xml (The system cannot find the file specified)

Version:

WLS 10.3.6 and ADR 11.1.1.7.0 on Windows

Issue:

Getting errors when Managed Server was tried to start from weblogic console URL. Nodemanager was up and its status in console was Active. The same Managed Server can be started up using startManagedserver.cmd

Error:

<Nov 17, 2014 11:45:48 PM MST> <Warning> <Security> <BEA-090904> <The JPS or JRF jar is present in the classpath, but no system property supplied for "oracle.security.jps.config" and "oracle.domain.config.dir". The server will start without setting OPSS java policy provider.> 
Nov 17, 2014 11:45:50 PM oracle.security.jps.internal.config.xml.XmlConfigurationFactory initDefaultConfiguration
SEVERE: java.io.FileNotFoundException: .\config\jps-config.xml (The system cannot find the file specified)
Nov 17, 2014 11:45:50 PM oracle.security.jps.internal.jaas.LoginConfigurationImpl <init>
SEVERE: .\config\jps-config.xml (The system cannot find the file specified)

oracle.security.jps.config.JpsConfigurationException: .\config\jps-config.xml (The system cannot find the file specified)


Reason:

Java Policy Store (JPS) configuration file (jps-config.xml ) is used to configure Policy, Credential, Key Store, Login Module and User Identity Store location.

When the Managed Server is started using WebLogic Console or Node Manager (NMStart) it uses nodemanager.properties file. This file has a parameter named StartScriptEnabled, which decides whether to use the startup scripts

StartScriptEnabled=false means NOT to use the startup scripts (startWebLogic or startManagedWebLogic) to start a server.

The startup scripts  invoke setDomainEnv.sh that sets -Doracle.security.jps.config to location of $DOMAIN_HOME/config/fmwconfig/jps-config.xml.

Since in our case StartScriptEnabled was set to false, when invoked from nodemanager, it failed to load the jps config file.

Solution:


Set StartScriptEnabled=true in the nodemanager.properties file.
Restart the service

Instance Critical Process (SA00) died unexpectedly- Instance termination due to error 12752

Version:

Version:  12.1.0.2.0 - 64bit Production in Exadata (8-node RAC)

Issue:

RAC DB instance in 1 of 8 nodes not coming up. RAC DB instances are up and running in other 7 nodes.

Error:

Instance Critical Process (pid: 7, ospid: 72674, SA00) died unexpectedly
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Mon Dec 29 00:52:01 2014
System state dump requested by (instance=2, osid=72660 (PMON)), summary=[abnormal instance termination].
Mon Dec 29 00:52:02 2014
PMON (ospid: 72660): terminating the instance due to error 12752

Reason:

The DB instance was crashing out on startup due to shared memory issues in that node.

The process which was failing was SA00 which is a new process from 12C. 
SAnn is SGA Allocator which allocates the  rest of the SGA  in small  chunks. Only a small fraction of SGA is allocated during instance startup. rest allocation is done by SAnn background process from 12c.

Fix:

Free or make more shared memory in the node for the instance and startup the instance.