Recently I made two posts about the process for patch/upgrading your 21 Grid Infrastructure (GI) while the databases continue to be running. The first post shows how to do this using the GUI interface, and the second one show more details about the process for AFD/ACFS Kernel Driver Update. But here in this post, I will show how to do the Zero-Downtime Patch (zeroDowntimeGIPatching – ZDGIP) in silent mode.
This way to do the patch is important because allows you to automatize it. You can create your own script and call it (using Ansible, Puppet, Chief, etc.) to upgrade your servers (or farms) remotely.
Current Environment
The current environment is the same of the first post:
- OEL 8.4 Kernel 5.4.17-2102.201.3.el8uek.x86_64.
- Oracle GI 21c, version 21.3 with no one-off or patches installed.
- Oracle Database 21c, RU 21.5 (with OCW 21.5).
- TFA version is 21.4 (last available in March 2022).
- Nodes are not using Transparent HugePages.
- Is a RAC installation, with two nodes.
You can see the output for the info above in this txt file.
And I will apply the same RU 21.5 (21.5.0.0.220118) for GI which is patch 33531909.
Patch Process
The patch process is almost the same as the first post, the main change is the response file and the way to call the gridSetup.sh. So, for this reason, I recommend for you read the first (and second) post. Below you will see a quick review of previous steps and a focus on the new
Preparation
The preparation part is basically, the folder created for the new GI home, and unzip of the versions/files. You can see here the output from these steps:
################################################################################## # #Creating the new directories for GI at node01 # ################################################################################## [root@oel8n1-21c ~]# mkdir -p /u01/app/21.5.0.0/grid [root@oel8n1-21c ~]# chown grid /u01/app/21.5.0.0/grid [root@oel8n1-21c ~]# chgrp -R oinstall /u01/app/21.5.0.0/grid [root@oel8n1-21c ~]# ################################################################################## # #Creating the new directories for GI at node02 # ################################################################################## [root@oel8n2-21c ~]# mkdir -p /u01/app/21.5.0.0/grid [root@oel8n2-21c ~]# chown grid /u01/app/21.5.0.0/grid [root@oel8n2-21c ~]# chgrp -R oinstall /u01/app/21.5.0.0/grid [root@oel8n2-21c ~]# ################################################################################## # #Unzip the binaries as GRID user at node01 # ################################################################################## [root@oel8n1-21c ~]# su - grid [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ cd /u01/install/21.5 [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ unzip -q V1011504-01.zip -d /u01/app/21.5.0.0/grid [grid@oel8n1-21c 21.5]$ ################################################################################## # #Updating opatch with the last version for 21c - node01 # ################################################################################## [grid@oel8n1-21c 21.5]$ cp -R /u01/app/21.5.0.0/grid/OPatch ./OPatch-ORG [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ unzip -q p6880880_210000_Linux-x86-64.zip -d /u01/app/21.5.0.0/grid replace /u01/app/21.5.0.0/grid/OPatch/README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ /u01/app/21.5.0.0/grid/OPatch/opatch version OPatch Version: 12.2.0.1.28 OPatch succeeded. [grid@oel8n1-21c 21.5]$ ################################################################################## # #Continuing to unzip the files (now the patch 21.5) - node01 # ################################################################################## [grid@oel8n1-21c 21.5]$ pwd /u01/install/21.5 [grid@oel8n1-21c 21.5]$ [grid@oel8n1-21c 21.5]$ unzip -q p33531909_210000_Linux-x86-64.zip [grid@oel8n1-21c 21.5]$
So, you will need:
- The base version of GI 21.3.
- GI RU 21.5.
- The latest version of OPatch for 21c.
And you will have:
- 21.3 GI installed and running at /u01/app/21.0.0.0
- 21.3 GI unzipped at /u01/app/21.5.0.0
- OPatch 12.2.0.1.28 unzipped at /u01/app/21.5.0.0
- 21.5 RU unzipped at /u01/install/21.5/33531909
Response File
The most important part is creating the correct response file. I recommend you to read carefully the response file and the comments inside of it to comprehend each definition, and the values that you can specify. The template for the response file can be found in folder <Your GI HOME>/install/response/gridsetup.rsp. So, you copy this file to one temporary folder and edit it.
For the Zero-Downtime I think that two definitions are the most critical: oracle.install.option and oracle.install.crs.config.storageOption. I will explain why below.
oracle.install.option
It defines how the installation process will occur, it has some options:
#------------------------------------------------------------------------------- # Specify the installation option. # Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY or HA_SWONLY # - CRS_CONFIG : To register home and configure Grid Infrastructure for cluster # - HA_CONFIG : To register home and configure Grid Infrastructure for stand alone server # - UPGRADE : To register home and upgrade clusterware software of earlier release # - CRS_SWONLY : To register Grid Infrastructure Software home (can be configured for cluster # or stand alone server later) # - HA_SWONLY : To register Grid Infrastructure Software home (can be configured for stand # alone server later. This is only supported on Windows.) # - CRS_ADDNODE : To add more nodes to the cluster # - CRS_DELETE_NODE : To delete nodes to the cluster #-------------------------------------------------------------------------------
And for the Zero-Downtime patch I use/used the oracle.install.option as UPGRADE because this will install+upgrade the GI in just one step. Another possibility (that does not work – I explain below) is to use it as CRS_SWONLY (that will install only the new version), and later (after the root.sh) call gridSetup.sh using the option switchGridHome. They are the steps described in the MOS note Steps for Minimal Downtime Grid Infrastructure Out of Place ( OOP ) Patching using gridSetup.sh (Doc ID 2662762.1).
This will not work (I tested) because between the installation/root.sh/gridSetup.sh(with switchGridHome) the root.sh/rootupgrade.sh is not updated correctly at the second node and will fails. This occurs due to the Bug 33601195 – gridSetup -switchgridhome oop – root.sh Cenerated Incorrectly (Doc ID 33601195.8). The first discussion that I saw about the same error was here at UnknownDba post comments.
So, the only option that will work in this case is oracle.install.option=UPGRADE.
oracle.install.crs.config.storageOption
This option is simple and defines how the ASM works (In one additional post I will explain why and how the Zero-Downtime patch works). But with this parameter, you can have one idea. So, the main values are:
#------------------------------------------------------------------------------- # Specify the type of storage to use for Oracle Cluster Registry(OCR) and Voting # Disks files. Only applicable for Standalone cluster. # - FLEX_ASM_STORAGE # - CLIENT_ASM_STORAGE # - FILE_SYSTEM_STORAGE # - EXASCALE_STORAGE # # Option FILE_SYSTEM_STORAGE is only for STANDALONE cluster configuration. #-------------------------------------------------------------------------------
And to work, you will choose the oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE.
Used Response File
So, with these two parameters explained, I have my response file defined. The options that I used are:
[grid@oel8n1-21c ~]$ cat grid-OOP-ZDP.rsp oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v21.0.0 INVENTORY_LOCATION=/u01/app/oraInventory oracle.install.option=UPGRADE ORACLE_BASE=/u01/app/grid ORACLE_HOME=/u01/app/21.5.0.0/grid oracle.install.asm.OSDBA=asmdba oracle.install.asm.OSOPER=oper oracle.install.asm.OSASM=asmadmin oracle.install.crs.config.scanType=LOCAL_SCAN oracle.install.crs.config.ClusterConfiguration=STANDALONE oracle.install.crs.config.configureAsExtendedCluster=false oracle.install.crs.config.clusterName=oel8-21c oracle.install.crs.config.gpnp.configureGNS=false oracle.install.crs.config.autoConfigureClusterNodeVIP=false oracle.install.crs.config.clusterNodes=oel8n1-21c,oel8n2-21c oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE oracle.install.crs.config.useIPMI=false oracle.install.asm.diskGroup.name=SYSTEMDG oracle.install.asm.diskGroup.AUSize=1 oracle.install.asm.configureAFD=false oracle.install.crs.configureRHPS=false oracle.install.crs.config.ignoreDownNodes=false oracle.install.crs.configureGIMR=false oracle.install.crs.configureRemoteGIMR=false oracle.install.crs.RemoteGIMRCredFile= oracle.install.config.managementOption=NONE oracle.install.crs.rootconfig.executeRootScript=false [grid@oel8n1-21c ~]$
Above you can see what I selected and what is needed to do Zero-Downtime patch using silent mode with a response file. In your case, you need to adapt it for your needs like clusterNodes, clusterName, asm.DiskGroup.name (where the OCR/Vote disk are), AUSize, Oracle Base and Oracle Home, scanType, and configureGNS. They need to reflect your environment.
Patching
With the response file created we can call the gridSetup.sh with the following parameters:
- -silent -responseFile: Defines that will not use GUI and the response file to be used.
- -applyRU: This will apply the RU patch over the current home before starting the installation of the GI itself.
- -switchGridHome: This informs the install process that the GI will move from the older home to the new one. This is important because the oracle.install.option was defined as UPGRADE.
- -zeroDowntimeGIPatching: This is the new feature and informs the patch process that databases will continue to be run.
- -skipDriverUpdate: This set to not install the kernel modules directly. Again, my hint is: always assume that the GI patch will update the kernel drivers. So, always call using this option to avoid unexpected problems.
To call the patch we do (as GI owner):
[grid@oel8n1-21c grid]$ ./gridSetup.sh -waitforcompletion -silent -responseFile /home/grid/grid-OOP-ZDP.rsp -applyRU /u01/install/21.5/33531909 -switchGridHome -zeroDowntimeGIPatching -skipDriverUpdate
Preparing the home to patch...
Applying the patch /u01/install/21.5/33531909...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2022-03-27_05-06-26PM/installerPatchActions_2022-03-27_05-06-26PM.log
Launching Oracle Grid Infrastructure Setup Wizard...
You can find the log of this install session at:
/u01/app/oraInventory/logs/GridSetupActions2022-03-27_05-06-26PM/gridSetupActions2022-03-27_05-06-26PM.log
As a root user, execute the following script(s):
1. /u01/app/21.5.0.0/grid/root.sh
Execute /u01/app/21.5.0.0/grid/root.sh on the following nodes:
[oel8n1-21c, oel8n2-21c]
Run the scripts on the local node first. After successful completion, run the scripts in sequence on all other nodes.
Successfully Setup Software.
[grid@oel8n1-21c grid]$
Since it is in silent mode, no GUI, and at the end, we need to call root.sh in both nodes to complete. So, at the first node I called it:
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/root.sh Check /u01/app/21.5.0.0/grid/install/root_oel8n1-21c.oralocal_2022-03-27_17-26-00-572396484.log for the output of root script [root@oel8n1-21c ~]#
And (again) no output (since it is silent). And the log is:
[root@oel8n1-21c ~]# cat /u01/app/21.5.0.0/grid/install/root_oel8n1-21c.oralocal_2022-03-27_17-26-00-572396484.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/21.5.0.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/21.0.0.0/grid/lib:/u01/app/21.5.0.0/grid/lib:'
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n1-21c/crsconfig/rootcrs_oel8n1-21c_2022-03-27_05-26-42PM.log
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n1-21c/crsconfig/crs_prepatch_apply_oop_oel8n1-21c_2022-03-27_05-26-42PM.log
This software is "262" days old. It is a best practice to update the CRS home by downloading and applying the latest release update. Refer to MOS note 2731675.1 for more details.
Performing following verification checks ...
cluster upgrade state ...PASSED
OLR Integrity ...PASSED
Hosts File ...PASSED
Free Space: oel8n1-21c:/ ...PASSED
Free Space: oel8n2-21c:/ ...PASSED
Pre-check for Patch Application was successful.
CVU operation performed: stage -pre patch
Date: Mar 27, 2022 5:26:45 PM
Clusterware version: 21.0.0.0.0
CVU home: /u01/app/21.0.0.0/grid
Grid home: /u01/app/21.0.0.0/grid
User: grid
Operating system: Linux5.4.17-2102.201.3.el8uek.x86_64
2022/03/27 17:27:17 CLSRSC-347: Successfully unlock /u01/app/21.5.0.0/grid
2022/03/27 17:27:17 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n1-21c/crsconfig/crs_postpatch_apply_oop_oel8n1-21c_2022-03-27_05-27-18PM.log
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
CRS-1151: The cluster was successfully set to rolling patch mode.
2022/03/27 17:27:39 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2022/03/27 17:28:16 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2022/03/27 17:29:32 CLSRSC-4015: Performing install or upgrade action for Oracle Autonomous Health Framework (AHF).
2022/03/27 17:29:32 CLSRSC-4012: Shutting down Oracle Autonomous Health Framework (AHF).
2022/03/27 17:30:25 CLSRSC-4013: Successfully shut down Oracle Autonomous Health Framework (AHF).
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
2022/03/27 17:30:29 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
2022/03/27 17:30:45 CLSRSC-4003: Successfully patched Oracle Autonomous Health Framework (AHF).
[root@oel8n1-21c ~]#
As you can see above, everything is installed correctly and we can call the root.sh at node 02:
[root@oel8n2-21c ~]# /u01/app/21.5.0.0/grid/root.sh
Check /u01/app/21.5.0.0/grid/install/root_oel8n2-21c.oralocal_2022-03-27_17-33-17-771942165.log for the output of root script
[root@oel8n2-21c ~]#
[root@oel8n2-21c ~]# cat /u01/app/21.5.0.0/grid/install/root_oel8n2-21c.oralocal_2022-03-27_17-33-17-771942165.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/21.5.0.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/21.0.0.0/grid/lib:/u01/app/21.5.0.0/grid/lib:'
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n2-21c/crsconfig/rootcrs_oel8n2-21c_2022-03-27_05-34-24PM.log
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n2-21c/crsconfig/crs_prepatch_apply_oop_oel8n2-21c_2022-03-27_05-34-25PM.log
This software is "262" days old. It is a best practice to update the CRS home by downloading and applying the latest release update. Refer to MOS note 2731675.1 for more details.
Performing following verification checks ...
cluster upgrade state ...PASSED
OLR Integrity ...PASSED
Hosts File ...PASSED
Free Space: oel8n1-21c:/ ...PASSED
Free Space: oel8n2-21c:/ ...PASSED
Pre-check for Patch Application was successful.
CVU operation performed: stage -pre patch
Date: Mar 27, 2022 5:34:27 PM
Clusterware version: 21.0.0.0.0
CVU home: /u01/app/21.0.0.0/grid
Grid home: /u01/app/21.0.0.0/grid
User: grid
Operating system: Linux5.4.17-2102.201.3.el8uek.x86_64
2022/03/27 17:34:50 CLSRSC-347: Successfully unlock /u01/app/21.5.0.0/grid
2022/03/27 17:34:54 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel8n2-21c/crsconfig/crs_postpatch_apply_oop_oel8n2-21c_2022-03-27_05-34-54PM.log
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
CRS-1152: The cluster is in rolling patch mode.
CRS-4000: Command Start failed, or completed with errors.
2022/03/27 17:35:12 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2022/03/27 17:36:02 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2022/03/27 17:37:23 CLSRSC-4015: Performing install or upgrade action for Oracle Autonomous Health Framework (AHF).
2022/03/27 17:37:23 CLSRSC-4012: Shutting down Oracle Autonomous Health Framework (AHF).
2022/03/27 17:38:18 CLSRSC-4013: Successfully shut down Oracle Autonomous Health Framework (AHF).
Oracle Clusterware active version on the cluster is [21.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1452993786].
2022/03/27 17:38:38 CLSRSC-4003: Successfully patched Oracle Autonomous Health Framework (AHF).
Performing following verification checks ...
cluster upgrade state ...PASSED
Post-check for Patch Application was successful.
CVU operation performed: stage -post patch
Date: Mar 27, 2022 5:38:33 PM
Clusterware version: 21.0.0.0.0
CVU home: /u01/app/21.5.0.0/grid
Grid home: /u01/app/21.5.0.0/grid
User: grid
Operating system: Linux5.4.17-2102.201.3.el8uek.x86_64
2022/03/27 17:39:02 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@oel8n2-21c ~]#
And you can see below that database continued to be running while the entire CRS stack restarted (look at the time between the database process and the others):
################################################################################## # #Check what is running at node01 # ################################################################################## [root@oel8n1-21c ~]# date Sun Mar 27 17:40:45 CEST 2022 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep smon oracle 7144 1 0 16:45 ? 00:00:00 ora_smon_orcl21c1 root 84975 1 1 17:28 ? 00:00:09 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 85618 1 0 17:29 ? 00:00:00 asm_smon_+ASM1 root 112490 5284 0 17:40 pts/0 00:00:00 grep --color=auto smon [root@oel8n1-21c ~]# ps -ef |grep lsnr grid 85344 1 0 17:29 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 85423 1 0 17:29 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 95303 1 0 17:35 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit grid 95320 1 0 17:35 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit root 112530 5284 0 17:40 pts/0 00:00:00 grep --color=auto lsnr [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# date Sun Mar 27 17:40:51 CEST 2022 [root@oel8n1-21c ~]# ################################################################################## # #Check what is running at node02 # ################################################################################## [root@oel8n2-21c ~]# date Sun Mar 27 17:40:59 CEST 2022 [root@oel8n2-21c ~]# [root@oel8n2-21c ~]# ps -ef |grep smon oracle 6721 1 0 16:44 ? 00:00:00 ora_smon_orcl21c2 root 101956 1 1 17:36 ? 00:00:03 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 104624 1 0 17:37 ? 00:00:00 asm_smon_+ASM2 root 116445 6491 0 17:41 pts/0 00:00:00 grep --color=auto smon [root@oel8n2-21c ~]# ps -ef |grep lsnr grid 103478 1 0 17:37 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 103820 1 0 17:37 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 103875 1 0 17:37 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit root 116451 6491 0 17:41 pts/0 00:00:00 grep --color=auto lsnr [root@oel8n2-21c ~]# [root@oel8n2-21c ~]# date Sun Mar 27 17:41:08 CEST 2022 [root@oel8n2-21c ~]#
Post-Patch
For post-patch, you just need to upgrade the AFS/ACFS Kernel Drives. All the steps are covered in my additional post. Just remember that to do that you will need downtime (the reason that we called gridSetup.sh with the parameter skipDriverUpdate).
Conclusion
Patching the GI using the silent mode is quite easy, but for Zero-Downtime we need to take care of some details. As explained before the only working option for the response file is the UPGRADE for oracle.install.option. Just remember that using this you will need to call griSetup.sh with the parameter switchGridHome to allow the patch and Out-Of-Place (OOP) and Zero-Downtime Patch to occur together.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: Oracle Bloggers you should definitely like and subscribe