BEA-000286 Failed to invoke startup class "JRF Startup Class" ADF ManagedServer

Version:

WLS 10.3.6 and ADR 11.1.1.7.0 on Windows

Issue:

Getting errors when Managed Server was tried to start from weblogic console URL. Nodemanager was up and its status in console was Active. The same Managed Server can be started up using startManagedserver.cmd

Error:

<Dec 15, 2014 11:28:03 PM MST> <Critical> <WebLogicServer> <BEA-000286> <Failed to invoke startup class "JRF Startup Class", java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup
java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
Truncated. see log file for complete stacktrace

Reason:

From logs it was clear that issue is because of missing JRF classes in CLASSPATH.

JRF(Java Required Files) includes important libraries and capabilities that provides APIs that many OC4J (Oracle Container for Java, Oracle’s old JVM) applications depend upon.  JRF comes (under $MW_HOME/oracle_common) when you install Fusion Middleware component like ADF, SOA, WebCenter or Identity Management. 

Solution:

This class (JRF Startup) is in $MW_HOME\modules\oracle.jrf_11.1.1\jrf.jar

1) Add this path as part of WEBLOGIC CLASSPATH in commEnv.cmd

Class Path for Service is defined in $WLS_HOME\common\bin\commEnv.cmd (Eg: C:\Middleware\wlserver_10.3\common\bin). Add the entry $MW_HOME\modules\oracle.jrf_11.1.1\jrf.jar in WEBLOGIC_CLASSPATH in commEnv.cmd file under "set WEBLOGIC_CLASSPATH=" section


2) Restart the Managed Server.

PolicySet Invalid: WSM-06102 PolicyReference for 12c Enterprise Manager Cloud control

Version: 

OEM 12c Cloud control


Issue:

Standby DB creation is tried from EM using Availability - Add Standby Database.

On the submission of the job, it hangs and then fails on preparation phase.

The error received was 

PolicySet Invalid: WSM-06102 PolicyReference The policy reference URI "oracle/wss_username_token_client_policy" is not valid.


The error can be found from the emoms_pbs.trc files available under <OMS_Instance_Home>/em/<oms_name>/sysman/log directory.


Reason:

The weblogic and nodemanager credentails for the EM was reset recently. But few of the steps as part of password change was not done. 

Because of this even if we were able to  work with EM, the submission of jobs always failed or hung.

Solution:

Followed the Support Doc ID. 1450798.1. Redid all the steps exactly as mentioned in the document.

Reset the passwords back to the existing changed value.  After completion of all the above steps, tried the standby DB creation and it was successful.

Weblogic 10.3.6: java.io.IOException: Unable to backup file "weblogic-domain-binding.jar" during patching

Version:

Weblogic Server 10.3.6 in Windows 2008

Issue:

The bsu command to apply patch errors out

Patch applied was 18040640 (Patch Set Update 10.3.6.0.8).

Error:

Installing Patch ID: T5F1..Result: Failure
java.io.IOException: Unable to backup file C:\Middleware\wlserver_10.3\server\lib\schema\
weblogic-domain-binding.jar to C:\Middleware\wlserver_10.3\server\lib\schema\weblogic-dom
ain-binding.jar19755.bak

Reason: 

Few of the WLS files were already active during patching.

Patch was tried  straight after WLS installation. Nodemanager  was enabled as a service during installation and it was running.  Due to this the patching failed.

Action:

Stopped the nodemanager service from services.msc. Verified that no DLLs and files from the WLS Home is active using processExplorer tool. Reapplied the patch.

ODI-1400 and ODI-1545: Missing odi supervisor credentials in credential store map

Version:

ODI 11.1.1.7.0 64 bit

Issue:

The startManagedWeblogic log has following errors and OracleDIAgent status would be down in Enterprise Manager.

Unable to set the activation state to true for the application 'oraclediagent'.
weblogic.application.ModuleException: [HTTP:101216]Servlet: "AgentServlet" failed to preload on startup in Web application: "oraclediagent".

ODI-1400: Agent OracleDIAgent start failure: JDBC connection error occurs while connecting to the master repository.

Caused by: ODI-1545: Missing odi supervisor credentials in credential store map [oracle.odi.credmap] with key [SUPERVISOR].

Reason:

Security must be set up for the JAVA EE application to have access to the ODI repository. The entry will be created within the credential store, which will allow the JAVA EE Agent to authenticate itself to be able to use the required resources. The user is already set up in the ODI Security, in this case SUPERVISOR.

Action:

Execute Web Logic Scripting Tool (WLST), connect to our running Admin server and add the credential store

Steps:

1. Start WLST and connect

cd $ODI_HOME/common/bin

wlst.sh

wls:/offline> connect('weblogic','welcome1','t3://hostname.domain:7001')

2. Add the correct credential store for ODI Supervisor 

createCred (map="oracle.odi.credmap", key="SUPERVISOR", user="SUPERVISOR", password="SUNOPSIS", desc="ODI SUPERVISOR Credential")

Eg:
wls:/odi11g_domain/serverConfig> createCred(map="oracle.odi.credmap", key="SUPERVISOR", user="SUPERVISOR", password="sunopsis", desc="ODI SUPERVISOR Credential") 
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root. 
For more help, use help(domainRuntime)

3. Add the correct credential store for the  ODI Domain

createCred (map="oracle.odi.credmap", key="ODI Domain Name", user="weblogic", password="welcome1", desc="ODI-DOMAIN Credential")

Eg:
createCred(map="oracle.odi.credmap", key="odi11g_domain", user="weblogic", password="welcome1", desc="ODI-DOMAIN Credential")


4. Restart the Managed Server

./startManagedServer.sh odi_server1

Note:

To delete the credentials you can use deleteCred function from wlst

Eg:

deleteCred(map="oracle.odi.credmap", key="SUPERVISOR")

Oracle 12c PDB: user_dump_dest and trace file location

Version

Oracle DB 12c

USER_DUMP_DEST in PDB

The parameter user_dump_dest is deprecated in 12c. It is replaced by diagnostic_dest parameter from 11g.

Also this parameter cannot be set at PDB level. Even if it is set it doesnt have any impact and will be ignored.

Location of trace files

By default the trace files would get generated under the CDB folders under
diagnostic_dest.

diagnostic_dest                      string      /data/app/oracle

To identify issue the following query from PDB or CDB

SQL>  select value from v$diag_info where name='Diag Trace';

VALUE
--------------------------------------------------------------------------------
/data/app/oracle/diag/rdbms/orcl/orcl/trace

where orcl is my CDB name.


Verify the Path

Connect to the PDB instance and do a sample trace

SQL>  alter session set tracefile_identifier=trace_location;

SQL>  alter session set events '10046 trace name context forever, level 12';

SQL>  show parameter user_dump_dest

SQL>  select value from v$diag_info where name='Diag Trace';

SQL>  select value from v$diag_info where name='Default Trace File';

SQL>  exit

The trace file with the trace_location would be created under the folder /data/app/oracle/diag/rdbms/orcl/orcl/trace

DB Upgrade failure: wallet is not open

Scenario:

A database for which wallet is configured is being upgraded from version 11.2.0.3.0 to 11.2.0.4.0.
DB upgrade is done using DBUA

Issue:

During the initial phase of upgrade process when the DB is stopped and restarted, the upgrade process halts throwing "wallet is not open" message. 

Reason:

When the upgrade process is initiated from the new DB Home it doesn't have details to provide for the wallet file location, without which the database cannot be opened.

Solution:

Copy the sqlnet.ora file from the existing 11.2.0.3.0 Home to 11.2.0.4.0 Home's network/admin folder. Re-initiate the upgrade process

This will make the DB aware of the wallet path and the database can be opened easily

Opatch Error: The patch directory area must be a number.

Situation:

Applying  Oct 2014 SPU to Oracle DB software using opatch utility

DB Version: 11.2.0.4.0


Error:

The patch directory area must be a number.

ERROR: OPatch failed because of problems in patch area.


Solution:

Upgraded opatch to latest version.

After upgrade the patch application went smooth.

It can happen for other reasons also like  if  we try to apply a wrong release patch or instead of "opatch napply" "opatch apply" is  tried etc