Install

19c Combo OJVM-GI-RU

Overview

The following covers applying the OJVM-GI Release Update (RU) combo patch to a RAC database. Patch 30133178 is used in these examples (AKA October 2019).

This document is only intended for where the GI Home and the Database Homes are not shared and ACFS file system is not configured.

🠊 If you have a new unpatched 19c RAC perform this workaround before patching. 🠈

Combo Patch Fun Facts!

For both the Database RU and the OJVM RU

opatchauto brings down the services, applys the patch (and subpatches) then restarts the services on the node.
opatchauto only patches the individual node.
So you have to run it on all nodes.
opatchauto will patch both the GRID_HOME and ORACLE_HOME on the node.
So you don't have to explicitly execute it on the GI home then the DB home.
opatchauto updates the SQL in the database.
So you don't need to run datapatch -verbose afterward.
The GI RU opatchauto must be run from the GI Home as the root user.

OJVM RU

OJVM must be applied as a Nonrolling Patch unless specified otherwise.
The OJVM must be run from the DB Home as the root user.
To implement the OJVM PSU opatchauto apply is run on all nodes but the datapatch operation (AKA Load Modified SQL Files) is just run once from one node.

Prerequisites

Ensure you have the latest OPatch.
Confirm the Oracle inventory is OK.
$ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Download Combo patch (Doc ID 2118136.2):
1. What would you like to download? OJVM Update/PSU/Bundle Patches
2. Please select the version: 19.0.0.0
3. Download: OJVM + GI Update (30133178)
4. As root user copy patch .zip file to your patches dir.
  • cp /media/sf_sw/patches/19/Combo_OJVM-GI/p30133178_190000_Linux-x86-64.zip /u01/orasw/patches/
  • chown grid:oinstall /u01/orasw/patches/*
Run My Oracle Support's Conflict Checker to detect and resolve any conflicts.
Unzip the downloaded combo patch zip file.

su - grid
cd /u01/orasw/patches
unzip p30133178_190000_Linux-x86-64.zip

A sub directory with the matching patch number (30133178) will be created.

List included major patch sets.

cd /u01/orasw/patches/30133178
ls -l
30116789 (GI RU)
30128191 (OJVM RU)

There is a README.html in each top-level sub dir that identifies type (DB or OJVM).

One of the biggest reasons patches fail is lack of disk space in your ORACLE_HOME. Ensure you have 25gb or more free.

GI RU

Perform this on each node before doing OJVM RU.

 su -
 export PATH=$PATH:/u01/app/19.3.0.0.0/grid/OPatch
 cd /u01/app/19.3.0.0.0/grid/OPatch
 ./opatchauto apply /u01/orasw/patches/30133178/30116789

 Session Log
 /u01/app/19.3.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto<DateTime>.log
 Current Cmds Log
 /u01/app/19.3.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch<DateTime>_1.log

If this RAC created with only 1 node use the -nonrolling option: ./opatchauto apply /<PathToPatchNumber> -nonrolling.

🠊 Apply above GI RU on all nodes before continuing.

Check dba_registry

 COL version     FORMAT a10
 COL action      FORMAT a10
 COL status      FORMAT a10
 COL action_time FORMAT a30
 COL description FORMAT a65
 SELECT patch_id,patch_type,action,status,action_time,description FROM dba_registry_sqlpatch;

  PATCH_ID    VERSION     ACTION  STATUS   ACTION_TIME                    DESCRIPTION 
  ----------  ----------  ------  -------  -----------------------------  -------------------------------------------------------------- 
  30116789    19.x    APPLY   SUCCESS  DD-MON-YY nn.nn.nn.nnnnnn AA   DATABASE PATCH SET UPDATE 19.x.nnnnnnn

OJVM RU

After the GI RU completed for all nodes, open a new console to apply the OJVM RU from node 1 then other nodes.
This will bring down DB instance, apply OJVM PSU, then restart DB instance.

 su -
 export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0.0/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:/usr/ccs/bin
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto apply /u01/orasw/patches/30133178/30128191 -oh $ORACLE_HOME -nonrolling

OJVM Log Path: $ORACLE_HOME/cfgtoollogs/opatchautodb/systemconfig<DateTime>.log
For Oracle-Restart run the the OJVM without the -nonrolling option.
Oracle OJVM best practice info here.

Load Modified SQL Files Into the Database

Perform this on just node 1 once. Do not run on other nodes.
Ensure ORACLE_SID set for instance.

 a. su - oracle

 b. Create pfile From spfile
    oracle> sqlplus / as sysdba
    SQL> create pfile from spfile;
    oracle> ls -l $ORACLE_HOME/dbs

 c. Stop DB
    OS> srvctl stop database -d oradb

 d. Startup Upgrade
    oracle> sqlplus / as sysdba
    SQL> startup
    SQL> alter system set cluster_database=false scope=spfile;
    SQL> exit

    oracle> srvctl stop database -d oradb

    oracle> sqlplus / as sysdba
    SQL> startup upgrade;
    SQL> exit

 e. Run Datapatch 
    oracle> cd $ORACLE_HOME/OPatch
    oracle> ./datapatch -verbose

 f. Start DB Back to Normal Mode
    oracle> sqlplus / as sysdba
    SQL> alter system set cluster_database=true scope=spfile;
    SQL> shutdown
    SQL> exit

    oracle> srvctl start database -d oradb

 g. Test
    SELECT instance_name,status,database_status,active_state 
    FROM gv$instance ORDER BY instance_name;

    INSTANCE_NAME    STATUS       DATABASE_STATUS   ACTIVE_ST
    ---------------- ------------ ----------------- ---------
    oradb1	     OPEN	  ACTIVE             NORMAL
    oradb2	     OPEN	  ACTIVE             NORMAL

Make sure when you test that the database is not still in UPGRADE mode (AKA OPEN MIGRATE).
OJVM checks.

Post Combo Patch Actions

Ensure Oracle Inventory for the oracle home shows applied patches.
OS> $ORACLE_HOME/OPatch/opatch lsinventory|more
Perform database status checks.

   SELECT instance_name,status,database_status,active_state FROM gv$instance ORDER BY instance_name;
    INSTANCE_NAME    STATUS       DATABASE_STATUS   ACTIVE_ST 
    ---------------- ------------ ----------------- --------- 
    mydb1            OPEN         ACTIVE            NORMAL    
    mydb2            OPEN         ACTIVE            NORMAL

Check dba_registry

 COL version     FORMAT a10
 COL action      FORMAT a10
 COL status      FORMAT a10
 COL action_time FORMAT a30
 COL description FORMAT a65
 SELECT patch_id,patch_type,action,status,action_time,description FROM dba_registry_sqlpatch;

   PATCH_ID PATCH_TYPE ACTION     STATUS     ACTION_TIME                    DESCRIPTION 
  --------- ---------- ---------- ---------- ------------------------------ ---------------------------------------------------- 
   29517242 RU         APPLY      SUCCESS    31-JUL-19 02.55.38.683597 PM   Database Release Update : 19.3.0.0.190416 (29517242) 
   29834717 RU         APPLY      SUCCESS    02-AUG-19 10.39.07.471379 AM   Database Release Update : 19.4.0.0.190716 (29834717) 
   30128191 INTERIM   APPLY      SUCCESS    dd-MON-YY hh.mm.ss.nnnnnn AA  OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)

OJVM Status Check

References

This page has been derived from various MOS resources including: Doc ID 2285040.1, 3-20637596581, 3-16179254071, 3-16515328131, 3-17593686831, 3-18616506451.

Rollback Example Session

opatchauto rollback <UNZIPPED_PATCH_LOCATION>/<PATCH_NUMBER> [-nonrolling]

 -- Get Sub Patches
 cd /u01/orasw/patches/30133178 
 ls -l 
 30116789 (GI RU) 
 30128191 (OJVM RU)

 -- Rollback: RU
 export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0.0/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto rollback /u01/orasw/patches/30133178/30116789 -oh $ORACLE_HOME

 -- Rollback: OJVM
 export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0.0/dbhome_1
 export PATH=$ORACLE_HOME/bin:$PATH
 export PATH=$PATH:$ORACLE_HOME/OPatch
 cd $ORACLE_HOME/OPatch
 ./opatchauto rollback /u01/orasw/patches/30133178/30128191 -oh $ORACLE_HOME -nonrolling

 -- QC
 $ORACLE_HOME/OPatch/opatch lsinventory

Common Errors

opatchauto apply Results java.io.FileNotFoundException: <oraInventory>/ContentsXML/oui-patch.xml (Permission denied) Error in Non-OUI Nodes

Per Doc ID 2582139.1

Background

During 19.x Grid Infrastructure (GI) installation, the file 'oui-patch.xml' will be created under the central inventory directory on the OUI node (node where gridSetup.sh was invoked) but not on the remote nodes.

🠊 This is a bug. It WILL effect you unless you do the workaround. 🠈

Workaround

Workaround is to have the OUI-PATCH.XML file copied in the central inventory of Non-OUI node from the OUI node with permission for GRID_HOME owner and re-run opatchauto apply on Non-OUI node.

Example Session

Node 1 (as grid)

 cp /u01/app/oraInventory/ContentsXML/oui-patch.xml /media/sf_sw/patches/19/Combo_OJVM-GI/

Node 2 (as grid)

  cp /media/sf_sw/patches/19/Combo_OJVM-GI/oui-patch.xml /u01/app/oraInventory/ContentsXML
  chmod 660 /u01/app/oraInventory/ContentsXML/oui-patch.xml
  ls -la /u01/app/oraInventory/ContentsXML/oui-patch.xml

19c Combo OJVM-GI-RU

TOC

Overview

Combo Patch Fun Facts!

For both the Database RU and the OJVM RU

OJVM RU

Prerequisites

GI RU

Check dba_registry

OJVM RU

Load Modified SQL Files Into the Database

Post Combo Patch Actions

References

Rollback Example Session

Common Errors

opatchauto apply Results java.io.FileNotFoundException: <oraInventory>/ContentsXML/oui-patch.xml (Permission denied) Error in Non-OUI Nodes

Background

Workaround

Example Session