DB startup failure with ORA-27154, ORA-27300, ORA-27301 and ORA-27302

Version:

Oracle 12.1.0.1.0 64 bit on OEL 6.3

Issue:

Unable to start up 12c DB

Error:

SQL> startup;
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 17
ORA-27301: OS failure message: File exists
ORA-27302: failure occurred at: sskgpcreates

Troubleshoot:

Issue the command "sysresv" to view the currently allocated IPC resources

bash-4.1$ sysresv

IPC Resources for ORACLE_SID "orcl" :
Shared Memory
ID              KEY
No shared memory segments used
Semaphores:
ID              KEY
3964951         0xfbf59bfc
3997720         0xfbf59bfd
4030489         0xfbf59bfe
4063258         0xfbf59bff
4096027         0xfbf59c00
Oracle Instance not alive for sid "orcl"

We have to identify and cleanup any stray shared memory segments and semaphores still owned by the previous startup of Oracle. 

To get a list of shared memory and semaphore id's:

ipcs -a

To delete shared memory:

ipcrm -m <shared memory id>

To delete semaphores:

ipcrm -s <semaphore id>

Issue above commands as root user.

Commands:

[root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x1c8f7e64 229377     jtavares   640        14680064   0                       
0x00000000 262146     jtavares   640        1035993088 0                       
0x5d0c776c 950275     jtavares   640        4096       0                       
0x00000000 983044     jtavares   640        4096       0                       
0xa6728ac4 948862985  jtavares   640        4096       0                       
0x00000000 948895754  jtavares   640        4096       0                       
0x00000000 1111818251 jtavares   640        4096       0                       
0x00000000 1111851020 jtavares   640        4096       0                       
0x00000000 1111883789 jtavares   640        4096       0                       
0x49cc319c 1111916558 jtavares   640        12288      0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x2155cb64 163842     jtavares   640        151       
0x2155cb65 196611     jtavares   640        151       
0x2155cb66 229380     jtavares   640        151       
0x2155cb67 262149     jtavares   640        151       
0x2155cb68 294918     jtavares   640        151       
0x39fd8dd4 425991     jtavares   640        152       
0x39fd8dd5 458760     jtavares   640        152       

[root@devdbhost1 ~]# ipcrm -m 229377
[root@devdbhost1 ~]# ipcrm -m 262146
[root@devdbhost1 ~]# ipcrm -m 950275
[root@devdbhost1 ~]# ipcrm -m 983044
[root@devdbhost1 ~]# ipcrm -m 948862985
[root@devdbhost1 ~]# ipcrm -m 948895754
[root@devdbhost1 ~]# ipcrm -m 1111818251
[root@devdbhost1 ~]# ipcrm -m 1111851020
[root@devdbhost1 ~]# ipcrm -m 1111883789
[root@devdbhost1 ~]# ipcrm -m 1111916558

[root@devdbhost1 ~]# ipcrm -s 163842
[root@devdbhost1 ~]# ipcrm -s 196611
[root@devdbhost1 ~]# ipcrm -s 229380
[root@devdbhost1 ~]# ipcrm -s 262149
[root@devdbhost1 ~]# ipcrm -s 294918
[root@devdbhost1 ~]# ipcrm -s 425991
[root@devdbhost1 ~]# ipcrm -s 458760
[root@devdbhost1 ~]# ipcrm -s 491529
[root@devdbhost1 ~]# ipcrm -s 1081362
[root@devdbhost1 ~]# ipcrm -s 1114131
[root@devdbhost1 ~]# ipcrm -s 1146900
[root@devdbhost1 ~]# ipcrm -s 1179669
[root@devdbhost1 ~]# ipcrm -s 1212438
[root@devdbhost1 ~]# ipcrm -s 3964951
[root@devdbhost1 ~]# ipcrm -s 3997720
[root@devdbhost1 ~]# ipcrm -s 4030489
[root@devdbhost1 ~]# ipcrm -s 4063258
[root@devdbhost1 ~]# ipcrm -s 4096027
[root@devdbhost1 ~]# ipcrm -s 3506204
[root@devdbhost1 ~]# ipcrm -s 3538973
[root@devdbhost1 ~]# ipcrm -s 3571742



root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages 

Restarted DB successfully after this