Wednesday, January 21, 2015

java.sql.SQLException: IO Error: Got minus one from a read call and ORA-27300

Version:

ODI 11.1.1.6.0 with Oracle DB 11.2.0.4.0 in Linux X86_64

Issue:

ODI and Informatica connections to the target  DB is failing with SQL exception. New connections are not getting created.

Error:

ODI-1217: Session LOAD_ENCOUNTER_TYPE_DIMENSION (2576017) fails with return code 8000. 
ODI-1226: Step ENCOUNTER_TYPE_CODE fails after 1 attempt(s). 
ODI-1228: Task ENCOUNTER_TYPE_CODE (Variable) fails on the target  connection ORACLE_HDM. 
Caused By: java.sql.SQLException: IO Error: Got minus one from a read call 

From the database alert log file, we got that the actual cause of the error could be

Process P097 died, see its trace file
Process startup failed, error stack:
Errors in file /export/oracle/diag/rdbms/orcl/orcl/trace/orcl_psp0_29459.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Process P095 died, see its trace file

Analysis:

The error messages indicate that oracle has problem in forking required processes. This could be due to the limit set at the OS level per user.
There were 2 databases running on the same host and both had similar errors. The processes parameter were set at 2000 for each of them.

Following action items were executed as per Oracle support notes.

Increased nproc process limit to 16384 (/etc/security/limits.conf file)
Doubled the pid_max value to 72128 using command sysctl -w kernel.pid_max=72128
Restarted both the databases and listener

Still the problem persisted and no connections were possible.

Solution:

Manually edited the /etc/security/limits.d/90-nproc.conf file and updated the nproc value to 16384 from original value of 1024.
The entry in the above file explicitly overrides the settings in security.conf for the number of processes. It is a bug in some versions of Linux which enforces the old ulimit value of 1024 nproc's and overrides /etc/security/limits.conf custom settings.

The 90-nproc.conf file after editing will have entries as follows

*          soft    nproc     16384 
root       soft    nproc     unlimited

After editing its value to 16384 in 90-nproc.conf, the issue disappeared. It allowed the creation of more number of processes in the system

No comments:

Post a Comment