java.sql.SQLException: IO Error: Got minus one from a read call and ORA-27300

Version:

ODI 11.1.1.6.0 with Oracle DB 11.2.0.4.0 in Linux X86_64

Issue:

ODI and Informatica connections to the target  DB is failing with SQL exception. New connections are not getting created.

Error:

ODI-1217: Session LOAD_ENCOUNTER_TYPE_DIMENSION (2576017) fails with return code 8000. 
ODI-1226: Step ENCOUNTER_TYPE_CODE fails after 1 attempt(s). 
ODI-1228: Task ENCOUNTER_TYPE_CODE (Variable) fails on the target  connection ORACLE_HDM. 
Caused By: java.sql.SQLException: IO Error: Got minus one from a read call 

From the database alert log file, we got that the actual cause of the error could be

Process P097 died, see its trace file
Process startup failed, error stack:
Errors in file /export/oracle/diag/rdbms/orcl/orcl/trace/orcl_psp0_29459.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process P095 died, see its trace file

Analysis:

The error messages indicate that oracle has problem in forking required processes. This could be due to the limit set at the OS level per user.
There were 2 databases running on the same host and both had similar errors. The processes parameter were set at 2000 for each of them.

Following action items were executed as per Oracle support notes.

Increased nproc process limit to 16384 (/etc/security/limits.conf file)
Doubled the pid_max value to 72128 using command sysctl -w kernel.pid_max=72128
Restarted both the databases and listener

Still the problem persisted and no connections were possible.

Solution:

Manually edited the /etc/security/limits.d/90-nproc.conf file and updated the nproc value to 16384 from original value of 1024.
The entry in the above file explicitly overrides the settings in security.conf for the number of processes. It is a bug in some versions of Linux which enforces the old ulimit value of 1024 nproc's and overrides /etc/security/limits.conf custom settings.

The 90-nproc.conf file after editing will have entries as follows

*          soft    nproc     16384 
root       soft    nproc     unlimited

After editing its value to 16384 in 90-nproc.conf, the issue disappeared. It allowed the creation of more number of processes in the system

ORA-12520: TNS:listener could not find available handler for requested type of server

Version:

DB Version: 12.1.0.2.0

Issue: 

Connections to the DB are not happening either from sqlplus or from sqldeveloper. Tnsping command works fine

Error:

ORA-12520: TNS:listener could not find available handler for requested type of server. 

Reason:

When we log on to the DB host and connects directly without the tns service name, we get the actual error.

ORA-00020: maximum number of processes (300) exceeded

The error is due to Processes parameter value set in the database getting exceeded.

Fix:

Increase the parameter value in the database. We need to restart the DB to make the change effective since it is not dynamically modifiable.

SQL> alter system set processes=1000 scope=spfile;

System altered.

Shutdown and restart DB

java.io.FileNotFoundException: .\config\jps-config.xml (The system cannot find the file specified)

Version:

WLS 10.3.6 and ADR 11.1.1.7.0 on Windows

Issue:

Getting errors when Managed Server was tried to start from weblogic console URL. Nodemanager was up and its status in console was Active. The same Managed Server can be started up using startManagedserver.cmd

Error:

<Nov 17, 2014 11:45:48 PM MST> <Warning> <Security> <BEA-090904> <The JPS or JRF jar is present in the classpath, but no system property supplied for "oracle.security.jps.config" and "oracle.domain.config.dir". The server will start without setting OPSS java policy provider.> 
Nov 17, 2014 11:45:50 PM oracle.security.jps.internal.config.xml.XmlConfigurationFactory initDefaultConfiguration
SEVERE: java.io.FileNotFoundException: .\config\jps-config.xml (The system cannot find the file specified)
Nov 17, 2014 11:45:50 PM oracle.security.jps.internal.jaas.LoginConfigurationImpl <init>
SEVERE: .\config\jps-config.xml (The system cannot find the file specified)

oracle.security.jps.config.JpsConfigurationException: .\config\jps-config.xml (The system cannot find the file specified)


Reason:

Java Policy Store (JPS) configuration file (jps-config.xml ) is used to configure Policy, Credential, Key Store, Login Module and User Identity Store location.

When the Managed Server is started using WebLogic Console or Node Manager (NMStart) it uses nodemanager.properties file. This file has a parameter named StartScriptEnabled, which decides whether to use the startup scripts

StartScriptEnabled=false means NOT to use the startup scripts (startWebLogic or startManagedWebLogic) to start a server.

The startup scripts  invoke setDomainEnv.sh that sets -Doracle.security.jps.config to location of $DOMAIN_HOME/config/fmwconfig/jps-config.xml.

Since in our case StartScriptEnabled was set to false, when invoked from nodemanager, it failed to load the jps config file.

Solution:


Set StartScriptEnabled=true in the nodemanager.properties file.
Restart the service

Instance Critical Process (SA00) died unexpectedly- Instance termination due to error 12752

Version:

Version:  12.1.0.2.0 - 64bit Production in Exadata (8-node RAC)

Issue:

RAC DB instance in 1 of 8 nodes not coming up. RAC DB instances are up and running in other 7 nodes.

Error:

Instance Critical Process (pid: 7, ospid: 72674, SA00) died unexpectedly
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Mon Dec 29 00:52:01 2014
System state dump requested by (instance=2, osid=72660 (PMON)), summary=[abnormal instance termination].
Mon Dec 29 00:52:02 2014
PMON (ospid: 72660): terminating the instance due to error 12752

Reason:

The DB instance was crashing out on startup due to shared memory issues in that node.

The process which was failing was SA00 which is a new process from 12C. 
SAnn is SGA Allocator which allocates the  rest of the SGA  in small  chunks. Only a small fraction of SGA is allocated during instance startup. rest allocation is done by SAnn background process from 12c.

Fix:

Free or make more shared memory in the node for the instance and startup the instance.