Install

Combo OJVM-GI-DB Patch Set Update (PSU)

Overview

The following covers applying a OJVM-GI-DB PSU combo patch. Combo patches contain the OJVM PSU in the same zip file as GI PSU and DB PSU for ease of downloading. Run GI\DB PSU first on all nodes then the OJVM PSU on just one node. GI\DB PSU is AKA GI PSU on this page.

Always read the patch notes. These are just a cheat sheet for an environment where:

The GI home is not shared.
The database home is not shared.
ACFS is not used.

Patch 29252164 is used in these examples (AKA April 2019).

Combo Patch Fun Facts!

GI\DB PSU

opatchauto brings down the services, applys the patch (and subpatches) then restarts the services on the node.
opatchauto only patches the individual node.
So you have to run it on all nodes.
opatchauto will patch both the GRID_HOME and ORACLE_HOME on the node.
So you don't have to explicitly execute it on the GI home then the DB home.
opatchauto updates the SQL in the database.
So you don't need to run datapatch -verbose afterward.
The GI\DB PSU opatchauto must be run from the GI Home as the root user.

OJVM PSU

OJVM PSU must be applied as a Nonrolling Patch.
The OJVM PSU must be run from the DB Home as the root user.
To implement the OJVM PSU opatchauto apply is run on all nodes but the datapatch operation (AKA Load Modified SQL Files) is just run once from one node.

The above is derived from various MOS cases including: 3-16179254071, 3-16515328131, 3-17593686831, 3-18616506451.

Prerequisites

Ensure the ORACLE_SID for each instance set.
Download and install the latest OPatch.
Validate Oracle home for both the oracle and grid users.
$ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Download the latest PSU:
1. Go to MOS DB Proactive Patching Doc ID: 756671.1
2. From 756671.1, search on version - examples: Database 12.1.0.2, Database 19
3. Select latest PSU entry: Combo Of OJVM Component 12.1.0.2.nnn DB PSU + GI PSU12.1.0.2.nnn
4. Select Patch that matches your OS: Linux x86-64
5. Select Download then corresponding patch: pnnn_121020_Linux-x86-64.zip
As grid user copy patch .zip file to your patches dir on each node.
Example: cp /media/sf_sw/Oracle/patches/Combo_OJVM-GI-DB_PSU/p29252164_121020_Linux-x86-64.zip /u01/orasw/patches/
Run My Oracle Support's Conflict Checker tool to detect and resolve any conflicts.
If you have never used this go here.
Unzip the patch file.

su - grid

cd /u01/orasw/patches

unzip p29252164_121020_Linux-x86-64.zip

A sub directory with the matching patch number will be created.

One of the biggest reasons patches fail is lack of disk space in your home directories (oracle and grid). Ensure you have 25gb or more free.

List Sub Patches

You want to identify which is the GI PSU and which is the OJVM PSU.
There is a README.html in each top-level sub dir that identifies type (GI or OJVM).

Change to the top most patch dir and review README.html in each top-level sub dir.

 cd /u01/orasw/patches/29252164
 ls -l
 29176115 GI PSU
 29251241 OJVM PSU
 ...

GI PSU

Perform this on each node before doing OJVM PSU.

 su -
 export PATH=$PATH:/u01/app/12.1.0.2/grid/OPatch
 cd /u01/app/12.1.0.2/grid/OPatch
 ./opatchauto apply /u01/orasw/patches/29252164/29176115

 Session Log
 /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto<DateTime>.log
 Current Cmds Log
 /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch<DateTime>_1.log

If this RAC created with only 1 node use the -nonrolling option: ./opatchauto apply /<PathToPatchNumber> -nonrolling 🠊 Error Message Example.

🠊 Apply above GI PSU on all nodes before continuing.

Check dba_registry

 COL version     FORMAT a10
 COL action      FORMAT a10
 COL status      FORMAT a10
 COL action_time FORMAT a30
 COL description FORMAT a65
 SELECT patch_id,version,action,status,action_time,description FROM dba_registry_sqlpatch;

  PATCH_ID    VERSION     ACTION  STATUS   ACTION_TIME                    DESCRIPTION 
  ----------  ----------  ------  -------  -----------------------------  -------------------------------------------------------------- 
  29176115    12.1.0.2    APPLY   SUCCESS  22-APR-19 12.18.54.745812 PM   DATABASE PATCH SET UPDATE 12.1.0.2.190115

OJVM PSU

After the GI PSU completed for all nodes, open a new console to apply the OJVM PSU from node 1 then other nodes.
This will bring down DB instance, apply OJVM PSU, then restart DB instance.

 su -
 export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto apply /u01/orasw/patches/29252164/29251241 -oh $ORACLE_HOME -nonrolling

OJVM Log Path: $ORACLE_HOME/cfgtoollogs/opatchautodb/systemconfig<DateTime>.log
For Oracle-Restart run the the OJVM without the -nonrolling option.
Oracle OJVM best practice info here.

Load Modified SQL Files Into the Database

Perform this on just node 1 once. Do not run on other nodes.
Ensure ORACLE_SID set for instance.

 a. su - oracle

 b. Create pfile From spfile
    oracle> sqlplus / as sysdba
    SQL> create pfile from spfile;
    oracle> ls -l $ORACLE_HOME/dbs

 c. Stop DB
    OS> srvctl stop database -d oradb

 d. Startup Upgrade
    oracle> sqlplus / as sysdba
    SQL> startup
    SQL> alter system set cluster_database=false scope=spfile;
    SQL> exit

    oracle> srvctl stop database -d oradb

    oracle> sqlplus / as sysdba
    SQL> startup upgrade;
    SQL> exit

 e. Run Datapatch 
    oracle> cd $ORACLE_HOME/OPatch
    oracle> ./datapatch -verbose

 f. Start DB Back to Normal Mode
    oracle> sqlplus / as sysdba
    SQL> alter system set cluster_database=true scope=spfile;
    SQL> shutdown
    SQL> exit

    oracle> srvctl start database -d oradb

 g. Test
    SELECT instance_name,status,database_status,active_state 
    FROM gv$instance ORDER BY instance_name;

    INSTANCE_NAME    STATUS       DATABASE_STATUS   ACTIVE_ST
    ---------------- ------------ ----------------- ---------
    oradb1	     OPEN	  ACTIVE             NORMAL
    oradb2	     OPEN	  ACTIVE             NORMAL

Make sure when you test that the database is not still in UPGRADE mode (AKA OPEN MIGRATE).
OJVM checks.

Post Patch Actions

If any patches where identified during Patch Conflict Detection and Resolution step apply them now.
Ensure Oracle Inventory for grid and oracle homes shows applied patches.
OS> $ORACLE_HOME/OPatch/opatch lsinventory|more
Perform Status Checks.
Check dba_registry

 COL version     FORMAT a10
 COL action      FORMAT a10
 COL status      FORMAT a10
 COL action_time FORMAT a30
 COL description FORMAT a65
 SELECT patch_id,version,action,status,action_time,description FROM dba_registry_sqlpatch;

  PATCH_ID  VERSION  ACTION  STATUS  ACTION_TIME   DESCRIPTION 
  --------- -------- ------- ------- ------------- -------------------------------------------------------------- 
  29176115  12.1.0.2 APPLY   SUCCESS 22-APR-19 12.18.54.745812 PM   DATABASE PATCH SET UPDATE 12.1.0.2.190115 
  29251241  12.1.0.2 APPLY   SUCCESS 22-APR-19... Database PSU 12.1.0.2.190115, Oracle JavaVM Component (MonYY)

APPENDIX

Oracle-Restart

Oracle-Restart: GI PSU

Update OPatch.
Copy patch files to system.
Perform below steps.

 -- Get ORACLE_HOME for grid User
 su - grid
 printenv|grep ORA
 ORACLE_HOME=/u01/app/grid/product/12.1.0.2/grid

Use the ORACLE_HOME when setting PATH below.

 --Apply PSU
 su -
 export PATH=$PATH:/u01/app/grid/product/12.1.0.2/grid/OPatch
 cd /u01/app/grid/product/12.1.0.2/grid/OPatch
 ./opatchauto apply /u01/orasw/patches/29252164/29176115

Oracle-Restart: OJVM PSU

 -- Shutdown DB and Listener
 su - oracle
 srvctl stop database -d oradb_sb

 su - grid
 lsnrctl stop

 -- Run: opatch apply 
 su - oracle
 cd $ORACLE_HOME/OPatch
 ./opatch apply /u01/orasw/patches/29252164/29251241 -oh $ORACLE_HOME

 -- Startup DB and Listener
 srvctl start database -d oradb_sb
 Above will start the Listener.

 -- Run: datapatch
 su - oracle
 cd $ORACLE_HOME/OPatch
 ./datapatch -verbose

Locking and Unlocking Example

Lock

As root:

 export GRID_HOME=/u01/app/12.1.0.2/grid
 export PATH=$GRID_HOME/bin:$PATH
 cd /u01/app/12.2.0.1/grid/crs/install
 ./rootcrs.sh -unlock

 <<Apply Patches>>

Unlock

As root console used to lock:

 cd /u01/app/12.1.0.2/grid/crs/install
 ./rootcrs.sh -postpatch

Nonrolling Patch Example

Oracle JavaVM Component <vers> Database PSU used for this example.

 su -
 export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto apply /u01/orasw/patches/27726478/27475603 -oh $ORACLE_HOME -nonrolling

Disable Node from Restarting

su -
export GRID_HOME=/u01/app/12.1.0.2/grid
export PATH=$GRID_HOME/bin:$PATH
cd $GRID_HOME/bin

crsctl stop crs
crsctl disable crs

Rollback Example

As root: from OPatch dir:

./opatchauto rollback /u01/orasw/patches/27726478/27468957 -oh /u01/app/grid/product/12.1.0.2/grid
$ORACLE_HOME/OPatch/opatch lsinventory

Successful Patch Output Example

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:lnx01
RAC Home:/u01/app/oracle/product/12.1.0.2/dbhome_1
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/orasw/patches/27726478/27468957/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/orasw/patches/27726478/27468957/27338013
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /u01/orasw/patches/27726478/27468957/27338020
Log: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-48-33AM_1.log

Patch: /u01/orasw/patches/27726478/27468957/27338041
Log: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-48-33AM_1.log

Host:lnx01
CRS Home:/u01/app/12.1.0.2/grid
Version:12.1.0.2.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/orasw/patches/27726478/27468957/26983807
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-51-00AM_1.log

Patch: /u01/orasw/patches/27726478/27468957/27338013
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-51-00AM_1.log

Patch: /u01/orasw/patches/27726478/27468957/27338020
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-51-00AM_1.log

Patch: /u01/orasw/patches/27726478/27468957/27338041
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2018-06-20_08-51-00AM_1.log

OPatchauto session completed at Wed Jun 20 09:07:33 2018
Time taken to complete the session 24 minutes, 4 seconds

Rollback Example Session

 -- Get Sub Patches
 cd /u01/orasw/patches/28317214
 ls -l
 27923320 OJVM
 27967747 PSU

 -- Rollback: OJVM
 export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto rollback /u01/orasw/patches/28317214/27923320 -oh $ORACLE_HOME -nonrolling

 -- Rollback: PSU
 export ORACLE_HOME=/u01/app/12.1.0.2/grid
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto rollback /u01/orasw/patches/28317214/27967747 -oh $ORACLE_HOME [-nonrolling]

 -- QC
 $ORACLE_HOME/OPatch/opatch lsinventory

Rollback Example Session: Sub-Patches

 -- Determine Sub Patches to Rollback
 su - oracle
 $ORACLE_HOME/OPatch/opatch lsinventory

 Interim patches (2) :

 Patch  27762253     : applied on Wed Sep 05 14:40:03 EDT 2018
 Unique Patch ID:  22108407
 Patch description:  "OCW PATCH SET UPDATE 12.1.0.2.180717 (27762253)"
 ...
 Patch  27547329     : applied on Wed Sep 05 14:38:48 EDT 2018
 Unique Patch ID:  22280349
 Patch description:  "Database Patch Set Update : 12.1.0.2.180717 (27547329)"

 -- Determine Path To Subpatch Directoies
 cd /u01/orasw/patches/28317214
 ls -l 
   drwxr-xr-x 4 grid oinstall    67 May 29 14:24 27923320
   drwxr-xr-x 7 grid oinstall   143 Jul 16 07:10 27967747
 cd 27967747
 ls -l
  drwxr-xr-x  4 grid oinstall     48 Jul 16 07:03 26983807
  drwxr-xr-x 18 grid oinstall   4096 Jul 16 07:07 27547329
  drwxr-xr-x  5 grid oinstall     62 Jul 16 07:08 27762253
  drwxr-xr-x  5 grid oinstall     62 Jul 16 07:06 27762277

 cd 27762253
 pwd
  /u01/orasw/patches/28317214/27967747/27762253  <-Use THIS for patch path for 27762253 etc..

 -- With DB Home set (in this case):
 Ensure other nodes up then:
 ./opatchauto rollback /u01/orasw/patches/28317214/27967747/27762253 -oh $ORACLE_HOME -nonrolling
 ./opatchauto rollback /u01/orasw/patches/28317214/27967747/27547329 -oh $ORACLE_HOME -nonrolling

Oracle Best Practice on OJVM

Oracle recommends applying the latest OJVM PSU/DBBP/Update patch to ALL databases that have Oracle JavaVM present in the database, regardless of whether you are explicitly using it or not.

Even if Oracle JavaVM is not present in the database it is best practice to install the OJVM PSU/Update in case a new database is created in the ORACLE_HOME.

For more info on this see: Doc ID 2217053.1.

1 Node RAC: PSU Error Example

OPATCHAUTO-72141: Grid patching cannot be performed in rolling mode.
OPATCHAUTO-72141: Grid is installed only on a single node.
OPATCHAUTO-72141: Execute opatchauto in non-rolling mode.
OPatchAuto failed.

Combo OJVM-GI-DB Patch Set Update (PSU)

TOC

Overview

Combo Patch Fun Facts!

GI\DB PSU

OJVM PSU

Prerequisites

List Sub Patches

GI PSU

Check dba_registry

OJVM PSU

Load Modified SQL Files Into the Database

Post Patch Actions

APPENDIX

Oracle-Restart

Oracle-Restart: GI PSU

Oracle-Restart: OJVM PSU

Locking and Unlocking Example

Lock

Unlock

Nonrolling Patch Example

Disable Node from Restarting

Rollback Example

Successful Patch Output Example

Rollback Example Session

Rollback Example Session: Sub-Patches

Oracle Best Practice on OJVM

1 Node RAC: PSU Error Example