Unable to set the activation state to true for the application JDBC Data Source-rac8

Scenario:

One of the ManagedServer was going to Admin State whenever it was tried to start. From the log files it was clear that it is due to some kind of corruption/wrong configuration with one of the eight data sources.

"<Jul 31, 2013 6:30:29 AM PDT> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'ManServer1 JDBC Data Source-rac8'.
weblogic.application.ModuleException: at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:388) at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:227) at weblogic.application.internal.flow.DeploymentCallbackFlow$2.next(DeploymentCallbackFlow.java:541) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52) at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:175)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(7480293907351086147S:host02.mydomain:base_domain:ManServ1 null)/297    [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at ManServ1-rac7(ClusterableRemoteRef(7480293907351086147S:host02.mydomain:base_domain:ManServ1 [7480293907351086147S:host02.mydomain:bas_domain:ManServ1/296])/296        [weblogic.jdbc.common.internal.RemoteDataSource])"


Reason:

Permission issue with a jdbc config file for one of the data sources. During startup weblogic was unable to read the respective configuration file from $DOMAIN_HOME/config/jdbc location. Because of that during startup of the managed server it got updated to a different JNDI Name ( This we can see from Weblogic console page => Services => DataSources => Check for the failing data source name and its respective JNDI Name).


Solution:


Update the JNDI Name in the console and activate changes.

Then change the ownership of the corresponding .xml file in $DOMAIN_HOME/config/jdbc to the WLS S/W owner.

Stop the managed Server which is in Admin state and restart it.

It should be back to Running mode now

Unable to start the Weblogic AdminServer 11G: Server subsystem failed. Reason: java.lang.NullPointerException

Scenario:

Users were unable to start the AdminServer ( weblogic 10.3.6). The startup failed with following error

####<Jul 31, 2013 4:47:04 AM PDT> <Critical> <WebLogicServer> <host02> <AdminServer> <main> <<WLS Kernel>> <> <> <1375271224766> <BEA-000386> <Server subsystem failed. Reason: java.lang.NullPointerException 
java.lang.NullPointerException at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:571) at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:556) at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:575)  at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:556) at weblogic.management.internal.PendingDirectoryManager.getAllFiles(PendingDirectoryManager.java:413) at weblogic.management.provider.internal.EditAccessImpl.<init>(EditAccessImpl.java:165)

Reason:

One user stopped and restarted the Admin service as "root" user instead of the S/W owner (which was oracle). He also tried to create datasources for the application after starting up the weblogic Admin server.

Impact:

The action modified the ownership and permissions on many of the domain's folders to root. This was stopping the Admin server from coming up.

Tried few work around like renaming the AdminServer folder and then starting up, removing the replicas.prop file under AdminServer/data/ldap/conf directory and renaming WLS_DIAGNOSTICS000000.DAT file under AdminServer/data/store/diagnostics location. But nothing worked in this case.

Solution:

Check for the permission of "pending" folder under $DOMAIN_HOME. If it is under root move it back to oracle (or the original WLS S/W owner).

Also check whether we have any files under pending. If so we can rename them too

chown -R oracle:oinstall pending

This did the trick. After this I was able to start up the AdminServer.

handlerjavax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert in Weblogic cluster

Scenario: 

A WLS cluster configuration in exalogic with one Admin server and 6 managed servers split across 3 nodes.

Each node has its own node manager. The node manager is started using WLST script. Still in the weblogic console URL, the status is "Inactive"

The log files have following message

"NMProcess: WARNING: Uncaught exception in server handlerjavax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from server1.domainname - 10.200.200.200. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
NMProcess: javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from server1.domainname - 10.200.200.200. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected."


Work around:

In this case, Admin Server and the Node Manager are using default demo certificates. The node manager configuration is done using the fully qualified machine name and entries are there in /etc/hosts. After trying so many options to debug, finally decided to disable the Host Name Varification.

Edit both Nodemanager startup script and weblogic startup script and add following lines.

1. Nodemanager startup script under $WLS_HOME/wlserver_10.3/server/bin

Take a backup of startNodeManager.sh script and edit it

JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false"
export JAVA_OPTIONS

Add it between the "export CLASSPATH" line and cd "${NODEMGR_HOME}" line as shown below

export CLASSPATH
export PATH
JAVA_OPTIONS=${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
export JAVA_OPTIONS
cd "${NODEMGR_HOME}"

2. Similarly take a backup of startWeblogic.sh script under $DOMAIN_HOME/bin

and add following entry

JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.ignoreHostnameVerification=true"
export JAVA_OPTIONS

Add it between the SAVE_CLASSPATH and trap 'stopAll' line as shown below

CLASSPATH="${SAVE_CLASSPATH}"

SAVE_CLASSPATH=""

JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.ignoreHostnameVerification=true"
export JAVA_OPTIONS


trap 'stopAll' 1 2 3 15

Restart all services. The warning is gone now and the node manager status of all nodes when checked from console page changed to "Reachable"

DeploymentException: [J2EE:160149]Error while processing library references

Error:

DeploymentException: [J2EE:160149]Error while processing library references "Unresolved application library references, defined in weblogic-application.xml:-oracle.wsm.seedpolicies"


Scenario: 

Installed ADF 11.1.1.6 and extended the already existing WLS 10.3.6 domain. The existing domain has an Admin server and a cluster of 6 managed servers.

Issue: While trying to startup the managed servers, all are going to ADMIN mode.

Reason: The OWSM seed policy libraries are not deployed on the managed servers

The out file of managed servers have following information.

"ManServer5.log00005:weblogic.management.DeploymentException: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: oracle.wsm.seedpolicies, Specification-Version: 11.1.1, Implementation-Version: 11.1.1, exact-match: false]"



Fixing:

1. Take a backup of config.xml file under $DOMAIN_HOME/config as config.xml_orig

2. Use "vi" editor and search for string "oracle.wsm.seedpolicies". This would be under main tag  <app-deployment> and sub tag library.

The current entry have only AdminServer where it is deployed. Add all the managed servers here seperated by a comma as shown below

  <library>
    <name>oracle.wsm.seedpolicies#11.1.1@11.1.1</name>
    <target>AdminServer,ManServer1,ManServer2,ManServer3,ManServer4,ManServer5,ManServer6</target>
    <source-path>/u01/app/Oracle/fmw-prov/oracle_common/modules/oracle.wsm.policies_11.1.1/wsm-seed-policies.jar</source-path>
    <staging-mode>nostage</staging-mode>
  </library>

3. Save the file and quit editor

4. Restart managed servers one by one. There wont be any problem in starting up.

Another way would have been to select the deployment servers during the domain extension time itself.

ORA-01732 during RCU schema creation due to "_simple_view_merging" set to FALSE

Scenario: 

Repository Creation Utility (RCU 11.1.1.6.0) was invoked in an 11.2.0.3.0 database to create repository schemas for SOA Infra.

The schema creation was erroring out all the time because of ORA-01732 error. Gathered extended trace in RCU level using following command
export RCU_LOG_LEVEL=TRACE  (default is ERROR). Still the log files were of not much help

INCIDENT_ERROR rcu: oracle.sysman.assistants.rcu.backend.action.ActionThread::run: Failed to execute method:  Excepton:
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.sysman.assistants.rcu.backend.action.ActionThread.run(JavaAction.java:495)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLSyntaxErrorException: ORA-01732: data manipulation operation not legal on this view

Reason: 

Performance Team changed the value of an undocumented parameter, "_simple_view_merging" to FALSE (default is TRUE)

Impact:  

Setting the parameter _simple_view_merging to FALSE is equivalent to using the NO_MERGE hint against the view. Since we anticipated that it has to do some thing with the View merging, we concentrated on the parameters that were modified from its default value.
A support note (Doc ID 886790.1) too gave a hint on this direction.

Solution:

We then modified the parameter in one of the eight instances of the Exadata Full rack and succesfully created RCU schemas by pointing only to the instance where it was restarted. 

alter system set "_simple_view_merging"=TRUE SID='ORCL1';


The parameter was then modified in all RAC instances after performance team confirmed that there was no impact on performance after the change.      

ORA-00959 Error while trying to drop a tablespace

Scenario: 

A tablespace has been created in the database through a script using Mixed case.Now we want to drop the tablespace.

SQL> select tablespace_name,status from dba_tablespaces;

TABLESPACE_NAME                STATUS
--------------------------     ---------
SYSTEM                                ONLINE
SYSAUX                                ONLINE
UNDOTBS1                             ONLINE
TEMP                                    ONLINE
USERS                                  ONLINE
PVPRSpon_CDM_INDEX             ONLINE

SQL> drop tablespace PVPRSpon_CDM_INDEX;
drop tablespace PVPRSpon_CDM_INDEX
*
ERROR at line 1:
ORA-00959: tablespace 'PVPRSPON_CDM_INDEX' does not exist

We are unable to drop the tablespace even if it is available. The reason is while creating the tablespace the name was specified in mixed case within dowble quotes as follows

SQL> create tablespace "PVPRSpon_CDM_INDEX" ....

SQL> SELECT tablespace_name FROM dba_tablespaces WHERE LOWER(tablespace_name) = 'PVPRSPON_CDM_INDEX'
  2  ;

no rows selected

SQL> SELECT tablespace_name FROM dba_tablespaces WHERE UPPER(tablespace_name) = 'PVPRSPON_CDM_INDEX';

TABLESPACE_NAME
------------------------------
PVPRSpon_CDM_INDEX

Solution:

To drop the tablespace mention the tablespace name within double quotes as shown below.

SQL> drop tablespace "PVPRSpon_CDM_INDEX" INCLUDING CONTENTS;

Tablespace dropped.

Now the tablespace stands droped.