21c, Zero-Downtime Oracle Grid Infrastructure Patching – Silent Mode

Recently I made two posts about the process for patch/upgrading your 21 Grid Infrastructure (GI) while the databases continue to be running. The first post shows how to do this using the GUI interface, and the second one show more details about the process for AFD/ACFS Kernel Driver Update. But here in this post, I will show how to do the Zero-Downtime Patch (zeroDowntimeGIPatching – ZDGIP) in silent mode.

This way to do the patch is important because allows you to automatize it. You can create your own script and call it (using Ansible, Puppet, Chief, etc.) to upgrade your servers (or farms) remotely.

Current Environment

The current environment is the same of the first post:

  • OEL 8.4 Kernel 5.4.17-2102.201.3.el8uek.x86_64.
  • Oracle GI 21c, version 21.3 with no one-off or patches installed.
  • Oracle Database 21c, RU 21.5 (with OCW 21.5).
  • TFA version is 21.4 (last available in March 2022).
  • Nodes are not using Transparent HugePages.
  • Is a RAC installation, with two nodes.

You can see the output for the info above in this txt file.

And I will apply the same RU 21.5 (21.5.0.0.220118) for GI which is patch 33531909.

Patch Process

The patch process is almost the same as the first post, the main change is the response file and the way to call the gridSetup.sh. So, for this reason, I recommend for you read the first (and second) post. Below you will see a quick review of previous steps and a focus on the new 

Preparation

The preparation part is basically, the folder created for the new GI home, and unzip of the versions/files. You can see here the output from these steps:

##################################################################################
#
#Creating the new directories for GI at node01
#  
##################################################################################
[root@oel8n1-21c ~]# mkdir -p /u01/app/21.5.0.0/grid
[root@oel8n1-21c ~]# chown grid /u01/app/21.5.0.0/grid
[root@oel8n1-21c ~]# chgrp -R oinstall /u01/app/21.5.0.0/grid
[root@oel8n1-21c ~]#

##################################################################################
#
#Creating the new directories for GI at node02
#  
##################################################################################
[root@oel8n2-21c ~]# mkdir -p /u01/app/21.5.0.0/grid
[root@oel8n2-21c ~]# chown grid /u01/app/21.5.0.0/grid
[root@oel8n2-21c ~]# chgrp -R oinstall /u01/app/21.5.0.0/grid
[root@oel8n2-21c ~]#

##################################################################################
#
#Unzip the binaries as GRID user at node01
#  
##################################################################################
[root@oel8n1-21c ~]# su - grid
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$ cd /u01/install/21.5
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$ unzip -q V1011504-01.zip -d /u01/app/21.5.0.0/grid
[grid@oel8n1-21c 21.5]$

##################################################################################
#
#Updating opatch with the last version for 21c - node01
#  
##################################################################################
[grid@oel8n1-21c 21.5]$ cp -R /u01/app/21.5.0.0/grid/OPatch ./OPatch-ORG
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$ unzip -q p6880880_210000_Linux-x86-64.zip -d /u01/app/21.5.0.0/grid
replace /u01/app/21.5.0.0/grid/OPatch/README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$ /u01/app/21.5.0.0/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.28
OPatch succeeded.
[grid@oel8n1-21c 21.5]$

##################################################################################
#
#Continuing to unzip the files (now the patch 21.5) - node01
#  
##################################################################################
[grid@oel8n1-21c 21.5]$ pwd
/u01/install/21.5
[grid@oel8n1-21c 21.5]$
[grid@oel8n1-21c 21.5]$ unzip -q p33531909_210000_Linux-x86-64.zip
[grid@oel8n1-21c 21.5]$

So, you will need:

  • The base version of GI 21.3.
  • GI RU 21.5.
  • The latest version of OPatch for 21c.

And you will have:

  • 21.3 GI installed and running at /u01/app/21.0.0.0
  • 21.3 GI unzipped at /u01/app/21.5.0.0
  • OPatch 12.2.0.1.28 unzipped at /u01/app/21.5.0.0
  • 21.5 RU unzipped at /u01/install/21.5/33531909

Response File

The most important part is creating the correct response file. I recommend you to read carefully the response file and the comments inside of it to comprehend each definition, and the values that you can specify. The template for the response file can be found in folder <Your GI HOME>/install/response/gridsetup.rsp. So, you copy this file to one temporary folder and edit it.

For the Zero-Downtime I think that two definitions are the most critical: oracle.install.option and oracle.install.crs.config.storageOption. I will explain why below.

oracle.install.option

It defines how the installation process will occur, it has some options:

#-------------------------------------------------------------------------------
# Specify the installation option.
# Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY or HA_SWONLY
#   - CRS_CONFIG  : To register home and configure Grid Infrastructure for cluster
#   - HA_CONFIG   : To register home and configure Grid Infrastructure for stand alone server
#   - UPGRADE     : To register home and upgrade clusterware software of earlier release
#   - CRS_SWONLY  : To register Grid Infrastructure Software home (can be configured for cluster
#                   or stand alone server later)
#   - HA_SWONLY   : To register Grid Infrastructure Software home (can be configured for stand
#                   alone server later. This is only supported on Windows.)
#   - CRS_ADDNODE : To add more nodes to the cluster
#   - CRS_DELETE_NODE : To delete nodes to the cluster
#-------------------------------------------------------------------------------

And for the Zero-Downtime patch I use/used the oracle.install.option as UPGRADE because this will install+upgrade the GI in just one step. Another possibility (that does not work – I explain below) is to use it as CRS_SWONLY (that will install only the new version), and later (after the root.sh) call gridSetup.sh using the option switchGridHome. They are the steps described in the MOS note Steps for Minimal Downtime Grid Infrastructure Out of Place ( OOP ) Patching using gridSetup.sh (Doc ID 2662762.1).

This will not work (I tested) because between the installation/root.sh/gridSetup.sh(with switchGridHome) the root.sh/rootupgrade.sh is not updated correctly at the second node and will fails. This occurs due to the Bug 33601195 – gridSetup -switchgridhome oop – root.sh Cenerated Incorrectly (Doc ID 33601195.8). The first discussion that I saw about the same error was here at UnknownDba post comments.

So, the only option that will work in this case is oracle.install.option=UPGRADE

oracle.install.crs.config.storageOption

This option is simple and defines how the ASM works (In one additional post I will explain why and how the Zero-Downtime patch works). But with this parameter, you can have one idea. So, the main values are:

#-------------------------------------------------------------------------------
# Specify the type of storage to use for Oracle Cluster Registry(OCR) and Voting
# Disks files. Only applicable for Standalone cluster.
#   - FLEX_ASM_STORAGE
#   - CLIENT_ASM_STORAGE
#   - FILE_SYSTEM_STORAGE
#   - EXASCALE_STORAGE
#
# Option FILE_SYSTEM_STORAGE is only for STANDALONE cluster configuration.
#-------------------------------------------------------------------------------

And to work, you will choose the oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE.

Used Response File

So, with these two parameters explained, I have my response file defined. The options that I used are:

[grid@oel8n1-21c ~]$ cat grid-OOP-ZDP.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v21.0.0
INVENTORY_LOCATION=/u01/app/oraInventory
oracle.install.option=UPGRADE
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/21.5.0.0/grid
oracle.install.asm.OSDBA=asmdba
oracle.install.asm.OSOPER=oper
oracle.install.asm.OSASM=asmadmin
oracle.install.crs.config.scanType=LOCAL_SCAN
oracle.install.crs.config.ClusterConfiguration=STANDALONE
oracle.install.crs.config.configureAsExtendedCluster=false
oracle.install.crs.config.clusterName=oel8-21c
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.clusterNodes=oel8n1-21c,oel8n2-21c
oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE
oracle.install.crs.config.useIPMI=false
oracle.install.asm.diskGroup.name=SYSTEMDG
oracle.install.asm.diskGroup.AUSize=1
oracle.install.asm.configureAFD=false
oracle.install.crs.configureRHPS=false
oracle.install.crs.config.ignoreDownNodes=false
oracle.install.crs.configureGIMR=false
oracle.install.crs.configureRemoteGIMR=false
oracle.install.crs.RemoteGIMRCredFile=
oracle.install.config.managementOption=NONE
oracle.install.crs.rootconfig.executeRootScript=false
[grid@oel8n1-21c ~]$

Above you can see what I selected and what is needed to do Zero-Downtime patch using silent mode with a response file. In your case, you need to adapt it for your needs like clusterNodes, clusterName, asm.DiskGroup.name (where the OCR/Vote disk are), AUSize, Oracle Base and Oracle Home, scanType, and configureGNS. They need to reflect your environment.

Patching

With the response file created we can call the gridSetup.sh with the following parameters:

  • -silent -responseFile: Defines that will not use GUI and the response file to be used.
  • -applyRU: This will apply the RU patch over the current home before starting the installation of the GI itself.
  • -switchGridHome: This informs the install process that the GI will move from the older home to the new one. This is important because the oracle.install.option was defined as UPGRADE.
  • -zeroDowntimeGIPatching: This is the new feature and informs the patch process that databases will continue to be run.
  • -skipDriverUpdate: This set to not install the kernel modules directly. Again, my hint is: always assume that the GI patch will update the kernel drivers. So, always call using this option to avoid unexpected problems.

To call the patch we do (as GI owner):

[grid@oel8n1-21c grid]$ ./gridSetup.sh -waitforcompletion -silent -responseFile /home/grid/grid-OOP-ZDP.rsp -applyRU /u01/install/21.5/33531909 -switchGridHome -zeroDowntimeGIPatching -skipDriverUpdate
Preparing the home to patch...
Applying the patch /u01/install/21.5/33531909...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2022-03-27_05-06-26PM/installerPatchActions_2022-03-27_05-06-26PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2022-03-27_05-06-26PM/gridSetupActions2022-03-27_05-06-26PM.log

As a root user, execute the following script(s):
        1. /u01/app/21.5.0.0/grid/root.sh

Execute /u01/app/21.5.0.0/grid/root.sh on the following nodes:
[oel8n1-21c, oel8n2-21c]

Run the scripts on the local node first. After successful completion, run the scripts in sequence on all other nodes.

Successfully Setup Software.
[grid@oel8n1-21c grid]$

 

Since it is in silent mode, no GUI, and at the end, we need to call root.sh in both nodes to complete. So, at the first node I called it:

[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/root.sh
Check /u01/app/21.5.0.0/grid/install/root_oel8n1-21c.oralocal_2022-03-27_17-26-00-572396484.log for the output of root script
[root@oel8n1-21c ~]#

And (again) no output (since it is silent). And the log is:

[root@oel8n1-21c ~]# cat /u01/app/21.5.0.0/grid/install/root_oel8n1-21c.oralocal_2022-03-27_17-26-00-572396484.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/21.5.0.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/21.0.0.0/grid/lib:/u01/app/21.5.0.0/grid/lib:'
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n1-21c/crsconfig/rootcrs_oel8n1-21c_2022-03-27_05-26-42PM.log
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n1-21c/crsconfig/crs_prepatch_apply_oop_oel8n1-21c_2022-03-27_05-26-42PM.log
This software is "262" days old. It is a best practice to update the CRS home by downloading and applying the latest release update. Refer to MOS note 2731675.1 for more details.

 Performing following verification checks ...

   cluster upgrade state ...PASSED
   OLR Integrity ...PASSED
   Hosts File ...PASSED
   Free Space: oel8n1-21c:/ ...PASSED
   Free Space: oel8n2-21c:/ ...PASSED

 Pre-check for Patch Application was successful.

 CVU operation performed:      stage -pre patch
 Date:                         Mar 27, 2022 5:26:45 PM
 Clusterware version:          21.0.0.0.0
 CVU home:                     /u01/app/21.0.0.0/grid
 Grid home:                    /u01/app/21.0.0.0/grid
 User:                         grid
 Operating system:             Linux5.4.17-2102.201.3.el8uek.x86_64

2022/03/27 17:27:17 CLSRSC-347: Successfully unlock /u01/app/21.5.0.0/grid
2022/03/27 17:27:17 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n1-21c/crsconfig/crs_postpatch_apply_oop_oel8n1-21c_2022-03-27_05-27-18PM.log
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
CRS-1151: The cluster was successfully set to rolling patch mode.
2022/03/27 17:27:39 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2022/03/27 17:28:16 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2022/03/27 17:29:32 CLSRSC-4015: Performing install or upgrade action for Oracle Autonomous Health Framework (AHF).
2022/03/27 17:29:32 CLSRSC-4012: Shutting down Oracle Autonomous Health Framework (AHF).
2022/03/27 17:30:25 CLSRSC-4013: Successfully shut down Oracle Autonomous Health Framework (AHF).
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
2022/03/27 17:30:29 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
2022/03/27 17:30:45 CLSRSC-4003: Successfully patched Oracle Autonomous Health Framework (AHF).
[root@oel8n1-21c ~]#

As you can see above, everything is installed correctly and we can call the root.sh at node 02:

[root@oel8n2-21c ~]# /u01/app/21.5.0.0/grid/root.sh
Check /u01/app/21.5.0.0/grid/install/root_oel8n2-21c.oralocal_2022-03-27_17-33-17-771942165.log for the output of root script
[root@oel8n2-21c ~]#
[root@oel8n2-21c ~]# cat /u01/app/21.5.0.0/grid/install/root_oel8n2-21c.oralocal_2022-03-27_17-33-17-771942165.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/21.5.0.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/21.0.0.0/grid/lib:/u01/app/21.5.0.0/grid/lib:'
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n2-21c/crsconfig/rootcrs_oel8n2-21c_2022-03-27_05-34-24PM.log
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n2-21c/crsconfig/crs_prepatch_apply_oop_oel8n2-21c_2022-03-27_05-34-25PM.log
This software is "262" days old. It is a best practice to update the CRS home by downloading and applying the latest release update. Refer to MOS note 2731675.1 for more details.

 Performing following verification checks ...

   cluster upgrade state ...PASSED
   OLR Integrity ...PASSED
   Hosts File ...PASSED
   Free Space: oel8n1-21c:/ ...PASSED
   Free Space: oel8n2-21c:/ ...PASSED

 Pre-check for Patch Application was successful.

 CVU operation performed:      stage -pre patch
 Date:                         Mar 27, 2022 5:34:27 PM
 Clusterware version:          21.0.0.0.0
 CVU home:                     /u01/app/21.0.0.0/grid
 Grid home:                    /u01/app/21.0.0.0/grid
 User:                         grid
 Operating system:             Linux5.4.17-2102.201.3.el8uek.x86_64

2022/03/27 17:34:50 CLSRSC-347: Successfully unlock /u01/app/21.5.0.0/grid
2022/03/27 17:34:54 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n2-21c/crsconfig/crs_postpatch_apply_oop_oel8n2-21c_2022-03-27_05-34-54PM.log
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
CRS-1152: The cluster is in rolling patch mode.
CRS-4000: Command Start failed, or completed with errors.
2022/03/27 17:35:12 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2022/03/27 17:36:02 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2022/03/27 17:37:23 CLSRSC-4015: Performing install or upgrade action for Oracle Autonomous Health Framework (AHF).
2022/03/27 17:37:23 CLSRSC-4012: Shutting down Oracle Autonomous Health Framework (AHF).
2022/03/27 17:38:18 CLSRSC-4013: Successfully shut down Oracle Autonomous Health Framework (AHF).
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1452993786].
2022/03/27 17:38:38 CLSRSC-4003: Successfully patched Oracle Autonomous Health Framework (AHF).

 Performing following verification checks ...

   cluster upgrade state ...PASSED

 Post-check for Patch Application was successful.

 CVU operation performed:      stage -post patch
 Date:                         Mar 27, 2022 5:38:33 PM
 Clusterware version:          21.0.0.0.0
 CVU home:                     /u01/app/21.5.0.0/grid
 Grid home:                    /u01/app/21.5.0.0/grid
 User:                         grid
 Operating system:             Linux5.4.17-2102.201.3.el8uek.x86_64

2022/03/27 17:39:02 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@oel8n2-21c ~]#

And you can see below that database continued to be running while the entire CRS stack restarted (look at the time between the database process and the others):

##################################################################################
#
#Check what is running at node01
#  
##################################################################################
[root@oel8n1-21c ~]# date
Sun Mar 27 17:40:45 CEST 2022
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep smon
oracle      7144       1  0 16:45 ?        00:00:00 ora_smon_orcl21c1
root       84975       1  1 17:28 ?        00:00:09 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid       85618       1  0 17:29 ?        00:00:00 asm_smon_+ASM1
root      112490    5284  0 17:40 pts/0    00:00:00 grep --color=auto smon
[root@oel8n1-21c ~]# ps -ef |grep lsnr
grid       85344       1  0 17:29 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid       85423       1  0 17:29 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid       95303       1  0 17:35 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid       95320       1  0 17:35 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
root      112530    5284  0 17:40 pts/0    00:00:00 grep --color=auto lsnr
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# date
Sun Mar 27 17:40:51 CEST 2022
[root@oel8n1-21c ~]#
##################################################################################
#
#Check what is running at node02
#  
##################################################################################
[root@oel8n2-21c ~]# date
Sun Mar 27 17:40:59 CEST 2022
[root@oel8n2-21c ~]#
[root@oel8n2-21c ~]# ps -ef |grep smon
oracle      6721       1  0 16:44 ?        00:00:00 ora_smon_orcl21c2
root      101956       1  1 17:36 ?        00:00:03 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid      104624       1  0 17:37 ?        00:00:00 asm_smon_+ASM2
root      116445    6491  0 17:41 pts/0    00:00:00 grep --color=auto smon
[root@oel8n2-21c ~]# ps -ef |grep lsnr
grid      103478       1  0 17:37 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      103820       1  0 17:37 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      103875       1  0 17:37 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
root      116451    6491  0 17:41 pts/0    00:00:00 grep --color=auto lsnr
[root@oel8n2-21c ~]#
[root@oel8n2-21c ~]# date
Sun Mar 27 17:41:08 CEST 2022
[root@oel8n2-21c ~]#

Post-Patch

For post-patch, you just need to upgrade the AFS/ACFS Kernel Drives. All the steps are covered in my additional post. Just remember that to do that you will need downtime (the reason that we called gridSetup.sh with the parameter skipDriverUpdate).

Conclusion

Patching the GI using the silent mode is quite easy, but for Zero-Downtime we need to take care of some details. As explained before the only working option for the response file is the UPGRADE for oracle.install.option. Just remember that using this you will need to call griSetup.sh with the parameter switchGridHome to allow the patch and Out-Of-Place (OOP) and Zero-Downtime Patch to occur together.

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.

Leave a Reply

Your email address will not be published.