Thursday, March 27, 2014

ORA-27300 with ORA-00600: internal error code, arguments: [4194]

Error:

ORA-27300: OS system dependent operation:semctl failed with status: 22
ORA-00600: internal error code, arguments: [4194], [], [^B^@], [], [], [], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced


Platform:

Oracle 11.2.0.1.0 EE on Red Hat Enterprise Linux Server release 5.5 32 Bit

Issue:

Database instance is terminated. On trying to startup the database instance, it crashes after mount stats throwing the following error stack
SQL> startup
ORACLE instance started.
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 13023
Session ID: 387 Serial number: 5

The alert log pointed to following error messages
ORA-27300: OS system dependent operation:semctl failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgpwpost1
ORA-27303: additional information: semid = 131073
ORA-00600: internal error code, arguments: [4194], [], [^B^@], [], [], [], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
Doing block recovery for file 3 block 4069
No block recovery was needed
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x418D04E0] [PC:0x9BD1E9A, kgegpa()+36] [flags: 0x0, count: 1]
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x418D04E0] [PC:0x9BD1E9A, kgegpa()+36] [flags: 0x2, count: 2]
Wed Mar 26 05:39:40 2014
ORA-1092 : opitsk aborting process
Wed Mar 26 05:39:57 2014
ORA-1092 : opitsk aborting process

Cause:

ORA-27300 normally denotes failure at O/S level and ORA-27302  points to removed shared memory segments. So in this case some thing happened at O/S level with 
shared memory segments.

The second part of the error message is an internal error with first arguement [4194]
This error indicates that a mismatch has been detected between redo records and rollback (undo) records.

It happens if Oracle cannot validate that the next undo record to be added to the undo block is equal to current maximum number plus one.

Solution:

So first we have to identify and remove the shared memory segment associated with the failed instance and then fix the next part related to ORA-00600.
Use ipcs and ipcrm commands to identify and remove the shared memory segments as follows

0-oracle@DBServer=> ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 1540097    root      644        52         2                       
0x00000000 1572867    root      644        16384      2                       
0x00000000 1605636    root      644        268        2                       
0x5f08cb8c 1802245    oracle    660        524320768  0          

63-oracle@DBServer=> ipcrm -m 1802245

Then to fix the ORA-00600 error we need to create a second undo tablespace after leaving out the current undo.
Please refer to metalink Note 1428786.1 to have a detailed idea about the issue. Below given are the steps I tried and which worked for me.

1. Create pfile from spfile and shutdown the database
2. Edit pfile and set undo_management=manual. 
3. Startup the DB using pfile
4. Create new undo tablespace as shown in example below
   Create undo tablespace undotbs2 datafile '<NAME>' size 100m;
5. Drop old undo tablespace
   drop tablespace undotbs1 including contents and datafiles;
6. shutdown immediate
7. Edit pfile to change undo_management=auto and change the undo_tablespace parameter to new undo tablespace, undotbs2
8. startup nomount and  create spfile from pfile 
9. startup force. 

The reason we create a new undo tablespace first is to use new undo segment numbers that are higher then the current max number to be used.

2 comments:

  1. Thanks a lot, this blog entry saved my life. This solution is not available anywhere else i looked last 24 hours.

    ReplyDelete
  2. Just wanted to add that in my case it was not possible to login hence create a pfile in the first place. If anyone is in that situation because of the above error in the alert log then following the steps mentioned here - http://otn-world.blogspot.com.au/2013/10/startup-fails-with-ora-01012-not-logged.html will help to login to the system and create a pfile.

    Thanks.

    ReplyDelete