java.lang.IllegalArgumentException: Unable to use KSS for Demo Key Stores, KSS is unavailable..

Version:
======

WebLogic Server Version: 12.2.1.0.0
Enterprise Linux 6


Issue
=====

Weblogic domain "AdminServer" was not starting up due to the error mentioned in the post heading.


Analysis
=======

Admin server.out file had the following error stack.

AdminServer.out
---------------------
Could not load Logmanager "oracle.core.ojdl.logging.ODLLogManager"
java.lang.ClassNotFoundException: oracle.core.ojdl.logging.ODLLogManager
        at com.oracle.classloader.PolicyClassLoader.loadClass(PolicyClassLoader.java:267)
        at com.oracle.classloader.weblogic.LaunchClassLoader.loadClass(LaunchClassLoader.java:62)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.util.logging.LogManager$1.run(LogManager.java:195)
        at java.util.logging.LogManager$1.run(LogManager.java:181)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.util.logging.LogManager.<clinit>(LogManager.java:181)
        at java.util.logging.Handler.<init>(Handler.java:50)
        at java.util.logging.StreamHandler.<init>(StreamHandler.java:118)
        at com.oracle.classloader.log.LogStreamHandler.<init>(LogStreamHandler.java:58)
        at com.oracle.classloader.log.LogStreamHandler.<init>(LogStreamHandler.java:47)
        at com.oracle.classloader.log.LogStreamHandler.<init>(LogStreamHandler.java:30)
        at com.oracle.classloader.log.Logger.getLogHandler(Logger.java:135)
        at com.oracle.classloader.log.Logger.access$100(Logger.java:25)
        at com.oracle.classloader.log.Logger$BootstrapHandler.initialize(Logger.java:100)
        at com.oracle.classloader.log.Logger.initialize(Logger.java:71)
        at com.oracle.classloader.PolicyClassLoader.initializeLogging(PolicyClassLoader.java:82)
        at com.oracle.classloader.weblogic.LaunchClassLoader.loadClass(LaunchClassLoader.java:65)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
<Jul 24, 2018 11:39:18 AM GMT> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Jul 24, 2018 11:39:18 AM GMT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Jul 24, 2018 11:39:18 AM GMT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 25.51-b03 from Oracle Corporation.>
<Jul 24, 2018 11:39:18 AM GMT> <Info> <RCM> <BEA-2165021> <"ResourceManagement" is not enabled in this JVM. Enable "ResourceManagement" to use the WebLogic Server "Resource Consumption Management" feature. To enable "ResourceManagement", you must specify the following JVM options in the WebLogic Server instance in which the JVM runs: -XX:+UnlockCommercialFeatures -XX:+ResourceManagement.>
<Jul 24, 2018 11:39:18 AM GMT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct  6 10:05:47 PDT 2015 1721936>
<Jul 24, 2018 11:39:19 AM GMT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 9 exceptions.  They are:
1. weblogic.management.provider.internal.RuntimeAccessImpl$ParseException: [Management:141266]Parsing failure in config.xml: java.lang.IllegalArgumentException: Unable to use KSS for Demo Key Stores, KSS is unavailable..
2. java.lang.IllegalStateException: Unable to perform operation: create on weblogic.management.provider.internal.RuntimeAccessImpl
3. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.management.provider.internal.RuntimeAccessService


Reason:
=======

setDomainEnv.sh file was edited and an entries for a few additional JAVA_OPTIONS were added.
One particular line was ended with two apostrophes instead of one ("")
This made all the libraries missing when it was called.

Solution:
========

Removed the additional apostrophe and succesfully restarted services

[DeploymentService:290066]Error occurred while downloading files from admin server

Version
========

Weblogic 10.3.6
Enterprisse Linux 6

Error :
=======

While trying to deploy a file  and click save below given error was found



An error occurred during activation of changes, please see the log for details.
Message icon - Error Exception occured while downloading files
Message icon - Error [DeploymentService:290066]Error occurred while downloading files from admin server for deployment request "1,517,969,705,031". Underlying error is: "null"

Checking Admin Server Log
=====================

####<Feb 7, 2018 5:44:22 AM UTC> <Info> <Deployer> <mmvbaxaesgp0010.int.oracleindustry.com> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <85f6879dee15bd51:77664bc0:1616df7a653:-8000-000000000000008f> <1517982262166> <BEA-149038> <Initiating Task for Signal : [Deployer:149026]deploy application Signal on SignalServer.>
####<Feb 7, 2018 5:44:31 AM UTC> <Warning> <DeploymentService> <host1.oracle.com> <AdminServer> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <85f6879dee15bd51:77664bc0:1616df7a653:-8000-0000000000000090> <1517982271238> <BEA-290065> <Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id "1,517,982,262,115" from server "SignalServer". Exception is: "java.io.FileNotFoundException: /u01/app/Oracle/Install/App/relnotes/Release_Notes.pdf (Permission denied)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at weblogic.utils.jars.JarFileUtils.addFilesToJar(JarFileUtils.java:120)

Also so messages in log like

####<Feb 7, 2018 5:44:31 AM UTC> <Warning> <Deployer> <mmvbaxaesgp0010.int.oracleindustry.com> <AdminServer> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <85f6879dee15bd51:77664bc0:1616df7a653:-8000-0000000000000091> <1517982271450> <BEA-149078> <Stack trace for message 149004
weblogic.management.DeploymentException: Exception occured while downloading files

Reason
======
It was a case of improper permission settings at file level. While checking that particular file, it was identified that the ownership of the file was with root whereas the application owner was oracle.

Solution
======

Changed permission and solved

-rw-r-----. 1 root       root         188529 Nov 13 12:31 Release_Notes.pdf
 cd /u01/app/Oracle/Install/App/relnotes


Changed to
-rw-r-----. 1 oracle       oinstall        188529 Nov 13 12:31 Release_Notes.pdf

DB startup failure with ORA-27154, ORA-27300, ORA-27301 and ORA-27302

Version:

Oracle 12.1.0.1.0 64 bit on OEL 6.3

Issue:

Unable to start up 12c DB

Error:

SQL> startup;
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 17
ORA-27301: OS failure message: File exists
ORA-27302: failure occurred at: sskgpcreates

Troubleshoot:

Issue the command "sysresv" to view the currently allocated IPC resources

bash-4.1$ sysresv

IPC Resources for ORACLE_SID "orcl" :
Shared Memory
ID              KEY
No shared memory segments used
Semaphores:
ID              KEY
3964951         0xfbf59bfc
3997720         0xfbf59bfd
4030489         0xfbf59bfe
4063258         0xfbf59bff
4096027         0xfbf59c00
Oracle Instance not alive for sid "orcl"

We have to identify and cleanup any stray shared memory segments and semaphores still owned by the previous startup of Oracle. 

To get a list of shared memory and semaphore id's:

ipcs -a

To delete shared memory:

ipcrm -m <shared memory id>

To delete semaphores:

ipcrm -s <semaphore id>

Issue above commands as root user.

Commands:

[root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x1c8f7e64 229377     jtavares   640        14680064   0                       
0x00000000 262146     jtavares   640        1035993088 0                       
0x5d0c776c 950275     jtavares   640        4096       0                       
0x00000000 983044     jtavares   640        4096       0                       
0xa6728ac4 948862985  jtavares   640        4096       0                       
0x00000000 948895754  jtavares   640        4096       0                       
0x00000000 1111818251 jtavares   640        4096       0                       
0x00000000 1111851020 jtavares   640        4096       0                       
0x00000000 1111883789 jtavares   640        4096       0                       
0x49cc319c 1111916558 jtavares   640        12288      0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x2155cb64 163842     jtavares   640        151       
0x2155cb65 196611     jtavares   640        151       
0x2155cb66 229380     jtavares   640        151       
0x2155cb67 262149     jtavares   640        151       
0x2155cb68 294918     jtavares   640        151       
0x39fd8dd4 425991     jtavares   640        152       
0x39fd8dd5 458760     jtavares   640        152       

[root@devdbhost1 ~]# ipcrm -m 229377
[root@devdbhost1 ~]# ipcrm -m 262146
[root@devdbhost1 ~]# ipcrm -m 950275
[root@devdbhost1 ~]# ipcrm -m 983044
[root@devdbhost1 ~]# ipcrm -m 948862985
[root@devdbhost1 ~]# ipcrm -m 948895754
[root@devdbhost1 ~]# ipcrm -m 1111818251
[root@devdbhost1 ~]# ipcrm -m 1111851020
[root@devdbhost1 ~]# ipcrm -m 1111883789
[root@devdbhost1 ~]# ipcrm -m 1111916558

[root@devdbhost1 ~]# ipcrm -s 163842
[root@devdbhost1 ~]# ipcrm -s 196611
[root@devdbhost1 ~]# ipcrm -s 229380
[root@devdbhost1 ~]# ipcrm -s 262149
[root@devdbhost1 ~]# ipcrm -s 294918
[root@devdbhost1 ~]# ipcrm -s 425991
[root@devdbhost1 ~]# ipcrm -s 458760
[root@devdbhost1 ~]# ipcrm -s 491529
[root@devdbhost1 ~]# ipcrm -s 1081362
[root@devdbhost1 ~]# ipcrm -s 1114131
[root@devdbhost1 ~]# ipcrm -s 1146900
[root@devdbhost1 ~]# ipcrm -s 1179669
[root@devdbhost1 ~]# ipcrm -s 1212438
[root@devdbhost1 ~]# ipcrm -s 3964951
[root@devdbhost1 ~]# ipcrm -s 3997720
[root@devdbhost1 ~]# ipcrm -s 4030489
[root@devdbhost1 ~]# ipcrm -s 4063258
[root@devdbhost1 ~]# ipcrm -s 4096027
[root@devdbhost1 ~]# ipcrm -s 3506204
[root@devdbhost1 ~]# ipcrm -s 3538973
[root@devdbhost1 ~]# ipcrm -s 3571742



root@devdbhost1 ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x6c6c6536 0          root       600        4096       0                       
0x00000000 714702865  root       600        393216     1          dest         
0x00000000 714801172  root       600        393216     1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages 

Restarted DB successfully after this

bsu.sh: java.lang.OutOfMemoryError: GC overhead limit exceeded

Version:

Oracle Weblogic server 10.3.6 in OEL 64 bit

Issue:

July 2015 PSU (WLS PATCH SET UPDATE 10.3.6.0.12) on weblogic server is failing with "OutOfMemoryError". Patching is failing immediately after it is invoked.

Error:

[oracle@soahost1 bsu]$ ./bsu.sh -install -patch_download_dir=/home/oracle/middleware/utils/bsu/cache_dir -patchlist=EJUW -prod_dir=/home/oracle/middleware/wlserver_10.3

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:448)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:464)
        at com.bea.plateng.patch.dao.cat.PatchCatalog.getPatchDependencies(PatchCatalog.java:56)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getInvalidatedPatchMap(PatchCatalogHelper.java:1621)
        at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:436)
        at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
        at com.bea.plateng.patch.PatchSystem.setCacheDir(PatchSystem.java:201)
        at com.bea.plateng.patch.Patch.main(Patch.java:281)

Reason:

The default java memory arguments  provided in bsu.sh file are not sufficient

Solution:

Increase the Java memory values in bsu.sh file (under $MW_HOME/utils/bsu) by adjusting the value MEM_ARGS values.

By default the values will be

MEM_ARGS="-Xms256m -Xmx512m"

I increased it to 1024-1024 as follows

"-Xms1024m -Xmx1024m"

Save the bsu.sh file and re-execute the patching command.

TNS-12555: TNS:permission denied with TNS-00525

Version:

Oracle EE 12.1.0.1.0 in Linux 64 bit

Issue:

Listener is not coming up when it is tried to start. We always ends up getting the same error log.

Errors:

TNSLSNR for Linux: Version 12.1.0.1.0 - Production
System parameter file is /home/oracle/product/121010/network/admin/listener.ora
Log messages written to /home/oracle/diag/tnslsnr/hostdb1/listener121010/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hostdb1.idc.oracle.com)(PORT=1211)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1211)))
TNS-12555: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00525: Insufficient privilege for operation
   Linux Error: 1: Operation not permitted

Listener failed to start. See the error message(s) above...

Reason: 

In most cases this will happen when "root" user was used for previous start of listener.That is the reason why we are getting permission denied and Insufficient privilege in error stack.

Solution:

1. cd /var/tmp

2. Check the ownership of the cd /var/tmp/.oracle folder. It should be such a way that the owner is the <oracle> owner and the group '<dba>' group of the oracle user.

In this case .oracle was owned by root as shown below

drwxrwxrwt   2 root root  4096 Apr 28 16:59 .oracle


3. Change the user and group using the appropriate command:

I have used the following commands as root user

chown oracle .oracle
chgrp dba .oracle
chmod 777 .oracle

After the change the folder permissions are as follows

drwxrwxrwt   2 oemora dba   4096 Apr 28 17:04 .oracle

4. Start the TNS Listener.

It should start normally now.

ld.so.1: oracle: fatal: libclntshcore.so.12.1: open failed: No such file or directory

Version:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

OS :Oracle Solaris on SPARC

Issue:

After applying PSU 12.1.0.2.3, connection to DB to do the startup, via sqlplus was failing. Was not able to connect to DB via sqlplus. The environment variables were set via ".oraenv" command.

Error:

sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Fri Jun 12 07:15:23 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ld.so.1: oracle: fatal: libclntshcore.so.12.1: open failed: No such file or directory
ERROR:
ORA-12547: TNS:lost contact

Verification:

The file mentioned () was present under the $ORACLE_HOME/lib folder.

 ls -ltr $ORACLE_HOME/lib | grep libclntshcore

-rwxrwxr-x   1 oradba   oinstall 7275016 Jun 12 07:10 libclntshcore.so.12.1

lrwxrwxrwx   1 oradba   oinstall      21 Jun 12 07:10 libclntshcore.so -> libclntshcore.so.12.1

So it could be due to the location missing in the Library path variable.

Solution:

Set the environment variable, LD_LIBRARY_PATH to $ORACLE_HOME/lib.

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

After this able to connect to sqlplus without any issues.

Oracle Weblogic: Setting independant JVM values for Managed Server in setDomainEnv.sh

Version: 

WLS 10.3.6 or FMW Infrastructure 12.1.3

Requirement:

Set the managed server to start with 1GB heap size and 1GB permgen space.

Steps:

1. Open the setDomainEnv.sh (under $DOMAIN_HOME/bin) in a text editor like vi

2. Search the file for the line starting with "IF USER_MEM_ARGS"

    Search for the commented out line starting with "IF USER_MEM_ARGS the environment variable     is set" in the file

3. Add the memory settings for your managed server below that line.
   
    Add the required settings for your managed server. 

Eg: 

# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values

if [ "${SERVER_NAME}" == "PRD_ManServer1" ] ; then
     USER_MEM_ARGS="-Xms1024m -Xmx1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m"
export USER_MEM_ARGS
fi
f [ "${USER_MEM_ARGS}" != "" ] ; then
        MEM_ARGS="${USER_MEM_ARGS}"
        export MEM_ARGS
fi

where PRD_MasServer1 is the name of my managed server. I have set both Java heap and permgen space to 1024 (both low and high values) in the above example.

4. Save the file and quit. Restart the managed service

5. Verify the change

    Grep weblogic process to make sure the change is effective

Eg: 

ps -ef|grep weblogic

/java/bin/java -server -Xms1024m -Xmx1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m -Dweblogic.Name=PRD_ManServer1