The process to patch Exadata stack and software changed in the last years and it became easier. Now, with patchmgr to be used for all (database servers, storage cells, and switches) the process is much easier to control the steps. Here I will show the steps that are involved in this process.
Independent if it is ZDLRA or Exadata, the process for Engineering System is the same. So, this post can be used as a guide for the Exadata patch apply as well. In 2018 I already made a similar process about how to patch/upgrade Exadata to 18c (you can access here) and even made a partial/incomplete post for 12c in 2015.
The process will be very similar and can be done in rolling and non-rolling mode. In the first, the services continue to run and you don’t need to shutdown databases, but will take more time because the patchmgr applies server by server. At the second, you need to shutdown the entire GI and the patch is applied in parallel and will be faster.
Exadata Version for ZDLRA
The Exadata version that can be used for ZDLRA needs to be marked as supported by the ZDLRA team and needs to be compatible with the version that you are running. This compatibility matrix is found at note Zero Data Loss Recovery Appliance Supported Versions (Doc ID 1927416.1).
I posted how to upgrade ZDLRA here, and the first thing to download the correct version is discovering which version of ZDLRA you are running. To do that, the racli version is used:
[root@zerosita01 ~]# racli version
Recovery Appliance Version:
exadata image: 19.2.3.0.0.190621
rarpm version: ra_automation-19.2.1.1.1.202001-RELEASE.x86_64
rdbms version: RDBMS_19.3.0.0.190416DBRU_LINUX.X64_RELEASE
transaction :
zdlra version: ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE
[root@zerosita01 ~]#
With that, check what version is compatible at the matrix:
So, we can see that version 19.2.1.1.1.202001 is compatible with several Exadata version. Here, in this example, I will use the version 19.3.2.0.0.
Prechecks
Now, we start to patch whatever the Engineering System you are using. Exadata, ZDLRA, SuperCluster. The process follows almost the same to all.
I recommend doing some prechecks before start the patch itself. The first it checks the actual Exadata version that you are running at database servers, storage servers, and switches. Doing this you can check if you need to apply intermediate patches (usually for switches this can occurs) because you are in an old version.
[root@zerosita01 radump]# dcli -l root -g /root/cell_group "imageinfo" |grep "Active image version" zerocell01: Active image version: 19.2.3.0.0.190621 zerocell02: Active image version: 19.2.3.0.0.190621 zerocell03: Active image version: 19.2.3.0.0.190621 zerocell04: Active image version: 19.2.3.0.0.190621 zerocell05: Active image version: 19.2.3.0.0.190621 zerocell06: Active image version: 19.2.3.0.0.190621 [root@zerosita01 radump]# [root@zerosita01 radump]# [root@zerosita01 radump]# [root@zerosita01 radump]# dcli -l root -g /root/db_group "imageinfo" |grep "Image version" zerosita01: Image version: 19.2.3.0.0.190621 zerosita02: Image version: 19.2.3.0.0.190621 [root@zerosita01 radump]# [root@zerosita01 radump]# dcli -l root -g /root/ib_group "version" |grep version zdls-iba01: SUN DCS 36p version: 2.2.12-2 zdls-iba01: BIOS version: NUP1R918 zdls-ibb01: SUN DCS 36p version: 2.2.12-2 zdls-ibb01: BIOS version: NUP1R918 [root@zerosita01 radump]# dcli -l root -g /root/ib_group "version" |grep "36p" zdls-iba01: SUN DCS 36p version: 2.2.12-2 zdls-ibb01: SUN DCS 36p version: 2.2.12-2 [root@zerosita01 radump]#
After that, I recommend the restart of ILOM from database and storage (if you do for the switch, the entire switch will restart). This is needed to avoid fsck of ILOM filesystem due longtime online (in old machines like V2, X2, the FSCK can take a lot of time and trigger the timeout and rollback of patchmgr), and to show if some hardware error was not triggered yet:
[root@zerosita01 radump]# dcli -g /root/cell_group -l root 'ipmitool mc reset cold' zerocell01: Sent cold reset command to MC zerocell02: Sent cold reset command to MC zerocell03: Sent cold reset command to MC zerocell04: Sent cold reset command to MC zerocell05: Sent cold reset command to MC zerocell06: Sent cold reset command to MC [root@zerosita01 radump]# [root@zerosita01 radump]# [root@zerosita01 radump]# dcli -g /root/db_group -l root 'ipmitool mc reset cold' zerosita01: Sent cold reset command to MC zerosita02: Sent cold reset command to MC [root@zerosita01 radump]# [root@zerosita01 radump]# ###################################################### After some time ###################################################### [root@zerosita01 radump]# dcli -l root -g /root/cell_group "ipmitool sunoem led get" |grep ": SERVICE" zerocell01: SERVICE | OFF zerocell02: SERVICE | OFF zerocell03: SERVICE | OFF zerocell04: SERVICE | OFF zerocell05: SERVICE | OFF zerocell06: SERVICE | OFF [root@zerosita01 radump]# [root@zerosita01 radump]# [root@zerosita01 radump]# [root@zerosita01 radump]# dcli -l root -g /root/db_group "ipmitool sunoem led get" |grep ": SERVICE" zerosita01: SERVICE | OFF zerosita02: SERVICE | OFF [root@zerosita01 radump]#
As you can see, after the ILOM restart, there is no HW error (service leds are off).
Downloading the patch
ZDLRA
To download the Exadata patch, the correct place is the note Zero Data Loss Recovery Appliance Supported Versions (Doc ID 1927416.1). And we can match the ZDLRA version that we are using with the Exadata:
If the link does not work (it’s common) the patch can be downloaded at note 888828.1 from Exadata. Be careful to download the same patch numbers.
I recommend downloading the patches do /radump folder. If you want, you can create one specific folder for that:
[root@zerosita01 radump]# mkdir exa_stack [root@zerosita01 radump]# cd exa_stack/ [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/patchmgr/p21634633_193100_Linux-x86-64.zip ./ [root@zerosita01 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/dbnode/p30439212_193000_Linux-x86-64.zip ./ [root@zerosita01 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/switch/p30469711_193000_Linux-x86-64.zip ./ [root@zerosita01 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/storage/p30540336_193000_Linux-x86-64.zip ./ [root@zerosita01 exa_stack]#
And after download, we can unzip the patches from Infiniband/RCoE, Storage, and database patch manager. The patch from database server it is not needed to be unzipped:
[root@zerosita01 exa_stack]# unzip p21634633_193100_Linux-x86-64.zip Archive: p21634633_193100_Linux-x86-64.zip creating: dbserver_patch_19.191113/ ... ... inflating: dbserver_patch_19.191113/patchmgr_functions inflating: dbserver_patch_19.191113/imageLogger inflating: dbserver_patch_19.191113/ExaXMLNode.pm [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# unzip p30439212_193000_Linux-x86-64.zip Archive: p30439212_193000_Linux-x86-64.zip inflating: exadata_ol7_base_repo_19.3.2.0.0.191119.iso inflating: README.html [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# unzip p30469711_193000_Linux-x86-64.zip Archive: p30469711_193000_Linux-x86-64.zip creating: patch_switch_19.3.2.0.0.191119/ inflating: patch_switch_19.3.2.0.0.191119/nxos.7.0.3.I7.6.bin ... ... inflating: patch_switch_19.3.2.0.0.191119/patchmgr_functions [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# unzip p30540336_193000_Linux-x86-64.zip Archive: p30540336_193000_Linux-x86-64.zip creating: patch_19.3.2.0.0.191119/ inflating: patch_19.3.2.0.0.191119/dostep.sh ... ... inflating: patch_19.3.2.0.0.191119/19.3.2.0.0.191119.iso creating: patch_19.3.2.0.0.191119/linux.db.rpms/ inflating: patch_19.3.2.0.0.191119/README.html [root@zerosita01 exa_stack]#
Exadata
Specific for Exadata, the right place it is download from the note 888828.1. There, you can choose the desired version that you want.
After that, you can put in one specific folder and unzip (as shown for ZDLRA above) the files.
Shutdown ZDLRA
This step is needed just for ZDLRA to shutdown it (the “software” and the database). Just login with the sqlplus and execute dbms_ra.shutdown as rasys user:
root@zerosita01 exa_stack]# su - oracle Last login: Wed Mar 4 14:07:44 CET 2020 from 99.234.123.138 on ssh [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ sqlplus rasys/xxxxxx SQL*Plus: Release 19.0.0.0.0 - Production on Wed Mar 4 14:10:07 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Mar 04 2020 14:00:02 +01:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> SELECT state FROM ra_server; STATE ------------- ON SQL> exec dbms_ra.shutdown; PL/SQL procedure successfully completed. SQL> SELECT state FROM ra_server; STATE ------------- OFF SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ srvctl stop database -d zdlras [oracle@zerosita01 ~]$
Patching
Pre-Patch
I recommend using the screen to avoid that your session being killed by a firewall or network issues during the patch. With this, a session that simulates a console connection is created and you can reconnect it if needed. You can download from https://linux.oracle.com/.
[root@zerosita01 exa_stack]# rpm -Uvh /zfs/ZDLRA_PATCHING/19.2.1.1.1/screen-4.1.0-0.25.20120314git3c2946.el7.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:screen-4.1.0-0.25.20120314git3c29################################# [100%] [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# screen -L -RR Patch-Exadata-Stack [root@zerosita01 exa_stack]# pwd /radump/exa_stack [root@zerosita01 exa_stack]#
Another detail is unmounting all NSF that you are using (and remove/comment from fstab too to avoid mounting during the boot). If not, you will receive a warning about that. This is important for Exadata because you can have long mount time during the startup and this can force the rollback because the time to do the patch reaches the limit. If you don’t want to unmount (because you are patching in rolling mode), there is one flag that can be passed at patchmgr to avoid mount during the reboot:
[root@zerosita01 exa_stack]# umount /zfs
[root@zerosita01 exa_stack]#
SUMMARY OF WARNINGS AND ERRORS FOR zerosita01:
zerosita01: WARNING: Active network mounts found on this DB node.
zerosita01: The following known issues will be checked for but require manual follow-up:
zerosita01: (*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
It is needed to create files with the hostname for the switches, dbnodes, and storage servers.
1 – Shutdown
This step depends on the chosen mode that you are patching. If it is in rolling mode, you can skip. If no, continue with the shutdown.
Since the patch will be in non-rolling mode, it is needed to shutdown all the services running in the machine. To avoid any kind of service running, I shutdown the cluster services and CRS in all nodes:
[root@zerosita01 exa_stack]# /u01/app/19.0.0.0/grid/bin/crsctl stop cluster -all CRS-2673: Attempting to stop 'ora.crsd' on 'zerosita01' CRS-2673: Attempting to stop 'ora.crsd' on 'zerosita02' ... ... CRS-2673: Attempting to stop 'ora.diskmon' on 'zerosita02' CRS-2677: Stop of 'ora.diskmon' on 'zerosita02' succeeded [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# /u01/app/19.0.0.0/grid/bin/crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zerosita01' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zerosita01' ... ... CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zerosita01' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# ssh -q zerosita02 Last login: Wed Mar 4 14:15:09 CET 2020 from 99.234.123.138 on ssh Last login: Wed Mar 4 14:15:57 2020 from zerosita01.zero.flisk.net [root@zerosita02 ~]# [root@zerosita02 ~]# /u01/app/19.0.0.0/grid/bin/crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zerosita02' CRS-2673: Attempting to stop 'ora.mdnsd' on 'zerosita02' ... ... CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zerosita02' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# logout [root@zerosita01 exa_stack]#
2 – Patch Switches
Stopping cells services
This part of the process is not a requirement but can be done only in non-rolling executions. The idea is to stop all services running at storage to avoid IB port errors. And doing this, the “alerthistory” from cell doesn’t report error:
[root@zerosita01 patch_switch_19.3.2.0.0.191119]# dcli -l root -g /root/cell_group "cellcli -e list cell" zerocell01: zerocell01 online zerocell02: zerocell02 online zerocell03: zerocell03 online zerocell04: zerocell04 online zerocell05: zerocell05 online zerocell06: zerocell06 online [root@zerosita01 patch_switch_19.3.2.0.0.191119]# [root@zerosita01 patch_switch_19.3.2.0.0.191119]# [root@zerosita01 patch_switch_19.3.2.0.0.191119]# dcli -l root -g /root/cell_group "cellcli -e alter cell shutdown services all" zerocell01: zerocell01: Stopping the RS, CELLSRV, and MS services... zerocell01: The SHUTDOWN of services was successful. zerocell02: zerocell02: Stopping the RS, CELLSRV, and MS services... zerocell02: The SHUTDOWN of services was successful. zerocell03: zerocell03: Stopping the RS, CELLSRV, and MS services... zerocell03: The SHUTDOWN of services was successful. zerocell04: zerocell04: Stopping the RS, CELLSRV, and MS services... zerocell04: The SHUTDOWN of services was successful. zerocell05: zerocell05: Stopping the RS, CELLSRV, and MS services... zerocell05: The SHUTDOWN of services was successful. zerocell06: zerocell06: Stopping the RS, CELLSRV, and MS services... zerocell06: The SHUTDOWN of services was successful. [root@zerosita01 patch_switch_19.3.2.0.0.191119]#
After that, we can start the patch for Infiniband switches.
ibswitch_precheck
This step checks if it is possible to apply the patch over ibswitch. Some versions have requirements that need to be matched, and if not, we need to fix (one example is hosts file that is wrong or space for FS).
To call the process we enter in the folder that corresponds to an unzipped patch from switches (patch 30469711 in this scenario):
[root@zerosita01 exa_stack]# cd patch_switch_19.3.2.0.0.191119
[root@zerosita01 patch_switch_19.3.2.0.0.191119]#
[root@zerosita01 patch_switch_19.3.2.0.0.191119]#
[root@zerosita01 patch_switch_19.3.2.0.0.191119]# ./patchmgr -ibswitches /root/ib_group -upgrade -ibswitch_precheck
2020-03-04 14:19:21 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-03-04 14:19:26 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-03-04 14:19:29 +0100 1 of 1 :Working: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started 2020-03-04 14:19:29 +0100 -----
[NOTE ] Log file at /radump/exa_stack/patch_switch_19.3.2.0.0.191119/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( zdls-iba01 zdls-ibb01 )
[SUCCESS ] Verifying Network connectivity to zdls-iba01
[SUCCESS ] Verifying Network connectivity to zdls-ibb01
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to "zdls-iba01" in all Switches
[INFO ] ---------- Starting with InfiniBand Switch zdls-iba01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host zerosita01.zero.flisk.net from the InfiniBand Switch zdls-iba01.
[INFO ] Starting pre-update validation on zdls-iba01
[SUCCESS ] Verifying that /tmp has 150M in zdls-iba01, found 492M
[SUCCESS ] Verifying that / has 20M in zdls-iba01, found 26M
[SUCCESS ] NTP daemon is running on zdls-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 14:26:40
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-iba01
[SUCCESS ] Verifying that the patchmgr host zerosita01.zero.flisk.net is recognized on the InfiniBand Switch zdls-iba01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on zdls-iba01
[INFO ] Finished pre-update validation on zdls-iba01
[SUCCESS ] Pre-update validation on zdls-iba01
[SUCCESS ] Prereq check on zdls-iba01
[INFO ] ---------- Starting with InfiniBand Switch zdls-ibb01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host zerosita01.zero.flisk.net from the InfiniBand Switch zdls-ibb01.
[INFO ] Starting pre-update validation on zdls-ibb01
[SUCCESS ] Verifying that /tmp has 150M in zdls-ibb01, found 492M
[SUCCESS ] Verifying that / has 20M in zdls-ibb01, found 26M
[SUCCESS ] NTP daemon is running on zdls-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 14:27:13
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-ibb01
[SUCCESS ] Verifying that the patchmgr host zerosita01.zero.flisk.net is recognized on the InfiniBand Switch zdls-ibb01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on zdls-ibb01
[INFO ] Finished pre-update validation on zdls-ibb01
[SUCCESS ] Pre-update validation on zdls-ibb01
[SUCCESS ] Prereq check on zdls-ibb01
[SUCCESS ] Overall status
----- InfiniBand switch update process ended 2020-03-04 14:20:54 +0100 -----
2020-03-04 14:20:54 +0100 1 of 1 :SUCCESS: Initiate pre-upgrade validation check on InfiniBand switch(es).
2020-03-04 14:20:54 +0100 :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -upgrade -ibswitch_precheck
2020-03-04 14:20:54 +0100 :INFO : upgrade attempted on nodes in file /root/ib_group: [zdls-iba01 zdls-ibb01]
2020-03-04 14:20:54 +0100 :INFO : For details, check the following files in /radump/exa_stack/patch_switch_19.3.2.0.0.191119:
2020-03-04 14:20:54 +0100 :INFO : - upgradeIBSwitch.log
2020-03-04 14:20:54 +0100 :INFO : - upgradeIBSwitch.trc
2020-03-04 14:20:54 +0100 :INFO : - patchmgr.stdout
2020-03-04 14:20:54 +0100 :INFO : - patchmgr.stderr
2020-03-04 14:20:54 +0100 :INFO : - patchmgr.log
2020-03-04 14:20:54 +0100 :INFO : - patchmgr.trc
2020-03-04 14:20:54 +0100 :INFO : Exit status:0
2020-03-04 14:20:55 +0100 :INFO : Exiting.
[root@zerosita01 patch_switch_19.3.2.0.0.191119]#
If the status is a success, we can continue to the next step.
upgrade
After the precheck, we can call the upgrade. The call is the same, just changing the parameter to upgrade:
[root@zerosita01 patch_switch_19.3.2.0.0.191119]# ./patchmgr -ibswitches /root/ib_group -upgrade
2020-03-04 14:30:46 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-03-04 14:30:51 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-03-04 14:30:54 +0100 1 of 1 :Working: Initiate upgrade of InfiniBand switches to 2.2.14-1. Expect up to 40 minutes for each switch
----- InfiniBand switch update process started 2020-03-04 14:30:54 +0100 -----
[NOTE ] Log file at /radump/exa_stack/patch_switch_19.3.2.0.0.191119/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( zdls-iba01 zdls-ibb01 )
[SUCCESS ] Verifying Network connectivity to zdls-iba01
[SUCCESS ] Verifying Network connectivity to zdls-ibb01
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1
[INFO ] Master Subnet Manager is set to "zdls-iba01" in all Switches
[INFO ] ---------- Starting with InfiniBand Switch zdls-iba01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host zerosita01.zero.flisk.net from the InfiniBand Switch zdls-iba01.
[INFO ] Starting pre-update validation on zdls-iba01
[SUCCESS ] Verifying that /tmp has 150M in zdls-iba01, found 492M
[SUCCESS ] Verifying that / has 20M in zdls-iba01, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch zdls-iba01
[SUCCESS ] NTP daemon is running on zdls-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 14:38:06
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-iba01
[SUCCESS ] Verifying that the patchmgr host zerosita01.zero.flisk.net is recognized on the InfiniBand Switch zdls-iba01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on zdls-iba01
[INFO ] Finished pre-update validation on zdls-iba01
[SUCCESS ] Pre-update validation on zdls-iba01
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on zdls-iba01
[INFO ] Starting upgrade on zdls-iba01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting zdls-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS ] Load firmware 2.2.14_1 onto zdls-iba01
[SUCCESS ] Verify that /conf/configvalid is set to 1 on zdls-iba01
[INFO ] Set SMPriority to 5 on zdls-iba01
[INFO ] Starting post-update validation on zdls-iba01
[SUCCESS ] Service opensmd is running on InfiniBand Switch zdls-iba01
[SUCCESS ] NTP daemon is running on zdls-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 14:54:47
[INFO ] /conf/configvalid is 1
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-iba01
[SUCCESS ] Execute plugin check for Post Patch on zdls-iba01
[INFO ] Finished post-update validation on zdls-iba01
[SUCCESS ] Post-update validation on zdls-iba01
[SUCCESS ] Update InfiniBand switch zdls-iba01 to 2.2.14_1
[INFO ] ---------- Starting with InfiniBand Switch zdls-ibb01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host zerosita01.zero.flisk.net from the InfiniBand Switch zdls-ibb01.
[INFO ] Starting pre-update validation on zdls-ibb01
[SUCCESS ] Verifying that /tmp has 150M in zdls-ibb01, found 492M
[SUCCESS ] Verifying that / has 20M in zdls-ibb01, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch zdls-ibb01
[SUCCESS ] NTP daemon is running on zdls-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 14:55:24
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-ibb01
[SUCCESS ] Verifying that the patchmgr host zerosita01.zero.flisk.net is recognized on the InfiniBand Switch zdls-ibb01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on zdls-ibb01
[INFO ] Finished pre-update validation on zdls-ibb01
[SUCCESS ] Pre-update validation on zdls-ibb01
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on zdls-ibb01
[INFO ] Starting upgrade on zdls-ibb01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting zdls-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS ] Load firmware 2.2.14_1 onto zdls-ibb01
[SUCCESS ] Verify that /conf/configvalid is set to 1 on zdls-ibb01
[INFO ] Set SMPriority to 5 on zdls-ibb01
[INFO ] Starting post-update validation on zdls-ibb01
[SUCCESS ] Service opensmd is running on InfiniBand Switch zdls-ibb01
[SUCCESS ] NTP daemon is running on zdls-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-03-04 Time:(HH:MM:SS) 15:12:05
[INFO ] /conf/configvalid is 1
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch zdls-ibb01
[SUCCESS ] Execute plugin check for Post Patch on zdls-ibb01
[INFO ] Finished post-update validation on zdls-ibb01
[SUCCESS ] Post-update validation on zdls-ibb01
[SUCCESS ] Update InfiniBand switch zdls-ibb01 to 2.2.14_1
[INFO ] InfiniBand Switches ( zdls-iba01 zdls-ibb01 ) updated to 2.2.14_1
[SUCCESS ] Overall status
----- InfiniBand switch update process ended 2020-03-04 15:05:48 +0100 -----
2020-03-04 15:05:48 +0100 1 of 1 :SUCCESS: Upgrade InfiniBand switch(es) to 2.2.14-1.
2020-03-04 15:05:48 +0100 :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -upgrade
2020-03-04 15:05:48 +0100 :INFO : upgrade attempted on nodes in file /root/ib_group: [zdls-iba01 zdls-ibb01]
2020-03-04 15:05:48 +0100 :INFO : For details, check the following files in /radump/exa_stack/patch_switch_19.3.2.0.0.191119:
2020-03-04 15:05:48 +0100 :INFO : - upgradeIBSwitch.log
2020-03-04 15:05:48 +0100 :INFO : - upgradeIBSwitch.trc
2020-03-04 15:05:48 +0100 :INFO : - patchmgr.stdout
2020-03-04 15:05:48 +0100 :INFO : - patchmgr.stderr
2020-03-04 15:05:48 +0100 :INFO : - patchmgr.log
2020-03-04 15:05:48 +0100 :INFO : - patchmgr.trc
2020-03-04 15:05:48 +0100 :INFO : Exit status:0
2020-03-04 15:05:48 +0100 :INFO : Exiting.
You have new mail in /var/spool/mail/root
[root@zerosita01 patch_switch_19.3.2.0.0.191119]# cd ..
[root@zerosita01 exa_stack]# dcli -l root -g /root/ib_group "version" |grep version
zdls-iba01: SUN DCS 36p version: 2.2.14-1
zdls-iba01: BIOS version: NUP1R918
zdls-ibb01: SUN DCS 36p version: 2.2.14-1
zdls-ibb01: BIOS version: NUP1R918
[root@zerosita01 exa_stack]#
As you can see, the patch uses the file “/root/ib_group” that contains the switches hostname and the patchmgr patched one by one. This mode to patch (one by one) is needed because of the architecture of the IB network. One switch is the “master” of the network.
The patch process is simple, copy the new version (through scp) to the switch and call the upgrade script. After that, reboot to load the new version.
Startup cells services
After the success of the patch, we can start the cell services:
[root@zerosita01 exa_stack]# dcli -l root -g /root/cell_group "cellcli -e alter cell startup services all" zerocell01: zerocell01: Starting the RS, CELLSRV, and MS services... zerocell01: Getting the state of RS services... running zerocell01: Starting CELLSRV services... zerocell01: The STARTUP of CELLSRV services was successful. zerocell01: Starting MS services... zerocell01: The STARTUP of MS services was successful. zerocell02: zerocell02: Starting the RS, CELLSRV, and MS services... zerocell02: Getting the state of RS services... running zerocell02: Starting CELLSRV services... zerocell02: The STARTUP of CELLSRV services was successful. zerocell02: Starting MS services... zerocell02: The STARTUP of MS services was successful. zerocell03: zerocell03: Starting the RS, CELLSRV, and MS services... zerocell03: Getting the state of RS services... running zerocell03: Starting CELLSRV services... zerocell03: The STARTUP of CELLSRV services was successful. zerocell03: Starting MS services... zerocell03: The STARTUP of MS services was successful. zerocell04: zerocell04: Starting the RS, CELLSRV, and MS services... zerocell04: Getting the state of RS services... running zerocell04: Starting CELLSRV services... zerocell04: The STARTUP of CELLSRV services was successful. zerocell04: Starting MS services... zerocell04: The STARTUP of MS services was successful. zerocell05: zerocell05: Starting the RS, CELLSRV, and MS services... zerocell05: Getting the state of RS services... running zerocell05: Starting CELLSRV services... zerocell05: The STARTUP of CELLSRV services was successful. zerocell05: Starting MS services... zerocell05: The STARTUP of MS services was successful. zerocell06: zerocell06: Starting the RS, CELLSRV, and MS services... zerocell06: Getting the state of RS services... running zerocell06: Starting CELLSRV services... zerocell06: The STARTUP of CELLSRV services was successful. zerocell06: Starting MS services... zerocell06: The STARTUP of MS services was successful. [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# dcli -l root -g /root/cell_group "cellcli -e list cell" zerocell01: zerocell01 online zerocell02: zerocell02 online zerocell03: zerocell03 online zerocell04: zerocell04 online zerocell05: zerocell05 online zerocell06: zerocell06 online [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# dcli -l root -g /root/cell_group "cellcli -e list alerthistory" [root@zerosita01 exa_stack]#
In the end, you can see that there is no error for cells.
3 – Patch Storage Cells
To patch the Storage Cell the process is similar, we use the patchmgr that it is inside of patch that was downloaded. The patch_19.3.2.0.0.191119 that we unzipped before.
[root@zerosita01 exa_stack]# ls -l total 7577888 drwxrwxr-x 3 root root 4096 Nov 13 16:11 dbserver_patch_19.191113 -rw-rw-r-- 1 root root 1505257472 Nov 19 19:23 exadata_ol7_base_repo_19.3.2.0.0.191119.iso -rw-r--r-- 1 root root 438739135 Mar 4 13:40 p21634633_193100_Linux-x86-64.zip -rw-r--r-- 1 root root 1460815642 Mar 4 13:46 p30439212_193000_Linux-x86-64.zip -rw-r--r-- 1 root root 2872128400 Mar 4 13:47 p30469711_193000_Linux-x86-64.zip -rw-r--r-- 1 root root 1474890484 Mar 4 13:48 p30540336_193000_Linux-x86-64.zip drwxrwxr-x 5 root root 4096 Nov 19 19:29 patch_19.3.2.0.0.191119 drwxrwxr-x 6 root root 4096 Mar 4 15:05 patch_switch_19.3.2.0.0.191119 -rw-rw-r-- 1 root root 264228 Nov 19 19:29 README.html -rw-r--r-- 1 root root 31912 Mar 4 15:05 screenlog.0 [root@zerosita01 exa_stack]#
cleanup
I always like to start the patch with the cleanup option for patchmgr. This connects in every cell and removes all possible old files (from the previous patch). It is a good start to avoid errors:
[root@zerosita01 exa_stack]# cd patch_19.3.2.0.0.191119 [root@zerosita01 patch_19.3.2.0.0.191119]# [root@zerosita01 patch_19.3.2.0.0.191119]# ./patchmgr -cells /root/cell_group -cleanup 2020-03-04 16:08:06 +0100 :Working: Cleanup 2020-03-04 16:08:08 +0100 :SUCCESS: Cleanup 2020-03-04 16:08:09 +0100 :SUCCESS: Completed run of command: ./patchmgr -cells /root/cell_group -cleanup 2020-03-04 16:08:09 +0100 :INFO : Cleanup attempted on nodes in file /root/cell_group: [zerocell01 zerocell02 zerocell03 zerocell04 zerocell05 zerocell06] 2020-03-04 16:08:09 +0100 :INFO : Current image version on cell(s) is: 2020-03-04 16:08:09 +0100 :INFO : zerocell01: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : zerocell02: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : zerocell03: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : zerocell04: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : zerocell05: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : zerocell06: 19.2.3.0.0.190621 2020-03-04 16:08:09 +0100 :INFO : For details, check the following files in /radump/exa_stack/patch_19.3.2.0.0.191119: 2020-03-04 16:08:09 +0100 :INFO : - <cell_name>.log 2020-03-04 16:08:09 +0100 :INFO : - patchmgr.stdout 2020-03-04 16:08:09 +0100 :INFO : - patchmgr.stderr 2020-03-04 16:08:09 +0100 :INFO : - patchmgr.log 2020-03-04 16:08:09 +0100 :INFO : - patchmgr.trc 2020-03-04 16:08:09 +0100 :INFO : Exit status:0 2020-03-04 16:08:09 +0100 :INFO : Exiting. [root@zerosita01 patch_19.3.2.0.0.191119]#
patch_check_prereq
The next is check the prereqs to do the patch (this test space and if the current version is compatible to do the direct upgrade):
[root@zerosita01 patch_19.3.2.0.0.191119]# ./patchmgr -cells /root/cell_group -patch_check_prereq 2020-03-04 16:09:22 +0100 :Working: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ... 2020-03-04 16:09:24 +0100 :SUCCESS: Check cells have ssh equivalence for root user. 2020-03-04 16:09:30 +0100 :Working: Initialize files. Up to 1 minute ... 2020-03-04 16:09:32 +0100 :Working: Setup work directory 2020-03-04 16:09:34 +0100 :SUCCESS: Setup work directory 2020-03-04 16:09:37 +0100 :SUCCESS: Initialize files. 2020-03-04 16:09:37 +0100 :Working: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ... 2020-03-04 16:09:52 +0100 :INFO : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes. 2020-03-04 16:09:53 +0100 :SUCCESS: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. 2020-03-04 16:09:53 +0100 :Working: Check space and state of cell services. Up to 20 minutes ... 2020-03-04 16:11:10 +0100 :SUCCESS: Check space and state of cell services. 2020-03-04 16:11:10 +0100 :Working: Check prerequisites on all cells. Up to 2 minutes ... 2020-03-04 16:11:13 +0100 :SUCCESS: Check prerequisites on all cells. 2020-03-04 16:11:13 +0100 :Working: Execute plugin check for Patch Check Prereq ... 2020-03-04 16:11:14 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0. 2020-03-04 16:11:14 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:11:14 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. 2020-03-04 16:11:14 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:11:14 +0100 :SUCCESS: No exposure to bug 17854520 with non-rolling patching 2020-03-04 16:11:14 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0. 2020-03-04 16:11:14 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:11:14 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216 2020-03-04 16:11:14 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0. 2020-03-04 16:11:14 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:11:14 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612 2020-03-04 16:11:14 +0100 :SUCCESS: No exposure to bug 22896791 with non-rolling patching 2020-03-04 16:11:14 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0. 2020-03-04 16:11:14 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:11:16 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315 2020-03-04 16:11:17 +0100 :SUCCESS: Execute plugin check for Patch Check Prereq. 2020-03-04 16:11:17 +0100 :Working: Check ASM deactivation outcome. Up to 1 minute ... 2020-03-04 16:11:28 +0100 :SUCCESS: Check ASM deactivation outcome. 2020-03-04 16:11:28 +0100 :Working: check if MS SOFTWAREUPDATE is scheduled. up to 5 minutes... 2020-03-04 16:11:29 +0100 :NO ACTION NEEDED: No cells found with SOFTWAREUPDATE scheduled by MS 2020-03-04 16:11:30 +0100 :SUCCESS: check if MS SOFTWAREUPDATE is scheduled 2020-03-04 16:11:32 +0100 :SUCCESS: Completed run of command: ./patchmgr -cells /root/cell_group -patch_check_prereq 2020-03-04 16:11:32 +0100 :INFO : patch_prereq attempted on nodes in file /root/cell_group: [zerocell01 zerocell02 zerocell03 zerocell04 zerocell05 zerocell06] 2020-03-04 16:11:32 +0100 :INFO : Current image version on cell(s) is: 2020-03-04 16:11:32 +0100 :INFO : zerocell01: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : zerocell02: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : zerocell03: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : zerocell04: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : zerocell05: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : zerocell06: 19.2.3.0.0.190621 2020-03-04 16:11:32 +0100 :INFO : For details, check the following files in /radump/exa_stack/patch_19.3.2.0.0.191119: 2020-03-04 16:11:32 +0100 :INFO : - <cell_name>.log 2020-03-04 16:11:32 +0100 :INFO : - patchmgr.stdout 2020-03-04 16:11:32 +0100 :INFO : - patchmgr.stderr 2020-03-04 16:11:32 +0100 :INFO : - patchmgr.log 2020-03-04 16:11:32 +0100 :INFO : - patchmgr.trc 2020-03-04 16:11:32 +0100 :INFO : Exit status:0 2020-03-04 16:11:32 +0100 :INFO : Exiting. [root@zerosita01 patch_19.3.2.0.0.191119]#
patch
After successfully prereqs check, we can do the patch. For rolling mode, you need to add the parameter “rolling” when you call the patchmgr. Doing that, it will patch one by one and just continue with the other cell if the resync from disks at ASM it is finished. And you need to be aware of the EX54 error. If you hit this error, the best is to apply in rolling mode.
The process to patch it is interesting because the patch itself copy the image in a different MD device (the one that it is not running), and change the boot to mount this MD. So, you do not upgrade the current system but just change the running image.
[root@zerosita01 patch_19.3.2.0.0.191119]# ./patchmgr -cells /root/cell_group -patch ******************************************************************************** NOTE Cells will reboot during the patch or rollback process. NOTE For non-rolling patch or rollback, ensure all ASM instances using NOTE the cells are shut down for the duration of the patch or rollback. NOTE For rolling patch or rollback, ensure all ASM instances using NOTE the cells are up for the duration of the patch or rollback. WARNING Do not interrupt the patchmgr session. WARNING Do not alter state of ASM instances during patch or rollback. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot cells or alter cell services during patch or rollback. WARNING Do not open log files in editor in write mode or try to alter them. NOTE All time estimates are approximate. ******************************************************************************** 2020-03-04 16:11:57 +0100 :Working: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ... 2020-03-04 16:11:59 +0100 :SUCCESS: Check cells have ssh equivalence for root user. 2020-03-04 16:12:05 +0100 :Working: Initialize files. Up to 1 minute ... 2020-03-04 16:12:07 +0100 :Working: Setup work directory 2020-03-04 16:12:41 +0100 :SUCCESS: Setup work directory 2020-03-04 16:12:44 +0100 :SUCCESS: Initialize files. 2020-03-04 16:12:44 +0100 :Working: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ... 2020-03-04 16:12:58 +0100 :INFO : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes. 2020-03-04 16:13:00 +0100 :SUCCESS: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. 2020-03-04 16:13:00 +0100 :Working: Check space and state of cell services. Up to 20 minutes ... 2020-03-04 16:14:15 +0100 :SUCCESS: Check space and state of cell services. 2020-03-04 16:14:15 +0100 :Working: Check prerequisites on all cells. Up to 2 minutes ... 2020-03-04 16:14:18 +0100 :SUCCESS: Check prerequisites on all cells. 2020-03-04 16:14:18 +0100 :Working: Copy the patch to all cells. Up to 3 minutes ... 2020-03-04 16:15:36 +0100 :SUCCESS: Copy the patch to all cells. 2020-03-04 16:15:38 +0100 :Working: Execute plugin check for Patch Check Prereq ... 2020-03-04 16:15:38 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0. 2020-03-04 16:15:38 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:15:38 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. 2020-03-04 16:15:38 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:15:38 +0100 :SUCCESS: No exposure to bug 17854520 with non-rolling patching 2020-03-04 16:15:38 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0. 2020-03-04 16:15:38 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:15:39 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216 2020-03-04 16:15:39 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0. 2020-03-04 16:15:39 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:15:39 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612 2020-03-04 16:15:39 +0100 :SUCCESS: No exposure to bug 22896791 with non-rolling patching 2020-03-04 16:15:39 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0. 2020-03-04 16:15:39 +0100 :INFO : Details in logfile /radump/exa_stack/patch_19.3.2.0.0.191119/patchmgr.stdout. 2020-03-04 16:15:41 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315 2020-03-04 16:15:42 +0100 :SUCCESS: Execute plugin check for Patch Check Prereq. 2020-03-04 16:15:42 +0100 :Working: check if MS SOFTWAREUPDATE is scheduled. up to 5 minutes... 2020-03-04 16:15:43 +0100 :NO ACTION NEEDED: No cells found with SOFTWAREUPDATE scheduled by MS 2020-03-04 16:15:44 +0100 :SUCCESS: check if MS SOFTWAREUPDATE is scheduled 2020-03-04 16:15:52 +0100 1 of 5 :Working: Initiate patch on cells. Cells will remain up. Up to 5 minutes ... 2020-03-04 16:15:58 +0100 1 of 5 :SUCCESS: Initiate patch on cells. 2020-03-04 16:15:58 +0100 2 of 5 :Working: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ... 2020-03-04 16:16:58 +0100 :INFO : Wait for patch pre-reboot procedures 2020-03-04 16:18:46 +0100 2 of 5 :SUCCESS: Waiting to finish pre-reboot patch actions. 2020-03-04 16:18:47 +0100 :Working: Execute plugin check for Patching ... 2020-03-04 16:18:47 +0100 :SUCCESS: Execute plugin check for Patching. 2020-03-04 16:18:47 +0100 3 of 5 :Working: Finalize patch on cells. Cells will reboot. Up to 5 minutes ... 2020-03-04 16:18:59 +0100 3 of 5 :SUCCESS: Finalize patch on cells. 2020-03-04 16:19:16 +0100 4 of 5 :Working: Wait for cells to reboot and come online. Up to 120 minutes ... 2020-03-04 16:20:16 +0100 :INFO : Wait for patch finalization and reboot 2020-03-04 16:51:25 +0100 4 of 5 :SUCCESS: Wait for cells to reboot and come online. 2020-03-04 16:51:26 +0100 5 of 5 :Working: Check the state of patch on cells. Up to 5 minutes ... 2020-03-04 16:52:40 +0100 5 of 5 :SUCCESS: Check the state of patch on cells. 2020-03-04 16:52:40 +0100 :Working: Execute plugin check for Pre Disk Activation ... 2020-03-04 16:52:41 +0100 :SUCCESS: Execute plugin check for Pre Disk Activation. 2020-03-04 16:52:41 +0100 :Working: Activate grid disks... 2020-03-04 16:52:42 +0100 :INFO : Wait for checking and activating grid disks 2020-03-04 16:54:27 +0100 :SUCCESS: Activate grid disks. 2020-03-04 16:54:31 +0100 :Working: Execute plugin check for Post Patch ... 2020-03-04 16:54:32 +0100 :SUCCESS: Execute plugin check for Post Patch. 2020-03-04 16:54:33 +0100 :Working: Cleanup 2020-03-04 16:55:08 +0100 :SUCCESS: Cleanup 2020-03-04 16:55:10 +0100 :SUCCESS: Completed run of command: ./patchmgr -cells /root/cell_group -patch 2020-03-04 16:55:10 +0100 :INFO : patch attempted on nodes in file /root/cell_group: [zerocell01 zerocell02 zerocell03 zerocell04 zerocell05 zerocell06] 2020-03-04 16:55:10 +0100 :INFO : Current image version on cell(s) is: 2020-03-04 16:55:10 +0100 :INFO : zerocell01: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : zerocell02: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : zerocell03: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : zerocell04: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : zerocell05: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : zerocell06: 19.3.2.0.0.191119 2020-03-04 16:55:10 +0100 :INFO : For details, check the following files in /radump/exa_stack/patch_19.3.2.0.0.191119: 2020-03-04 16:55:10 +0100 :INFO : - <cell_name>.log 2020-03-04 16:55:10 +0100 :INFO : - patchmgr.stdout 2020-03-04 16:55:10 +0100 :INFO : - patchmgr.stderr 2020-03-04 16:55:10 +0100 :INFO : - patchmgr.log 2020-03-04 16:55:10 +0100 :INFO : - patchmgr.trc 2020-03-04 16:55:10 +0100 :INFO : Exit status:0 2020-03-04 16:55:10 +0100 :INFO : Exiting. [root@zerosita01 patch_19.3.2.0.0.191119]#
Post-patch
After patch successfully we can check the running version and if every cell is up.
[root@zerosita01 patch_19.3.2.0.0.191119]# dcli -l root -g /root/cell_group "imageinfo" |grep "Active image version" zerocell01: Active image version: 19.3.2.0.0.191119 zerocell02: Active image version: 19.3.2.0.0.191119 zerocell03: Active image version: 19.3.2.0.0.191119 zerocell04: Active image version: 19.3.2.0.0.191119 zerocell05: Active image version: 19.3.2.0.0.191119 zerocell06: Active image version: 19.3.2.0.0.191119 [root@zerosita01 patch_19.3.2.0.0.191119]# [root@zerosita01 patch_19.3.2.0.0.191119]# dcli -l root -g /root/cell_group "cellcli -e list cell" zerocell01: zerocell01 online zerocell02: zerocell02 online zerocell03: zerocell03 online zerocell04: zerocell04 online zerocell05: zerocell05 online zerocell06: zerocell06 online [root@zerosita01 patch_19.3.2.0.0.191119]#
Exit the screen session and remove the screen package (to avoid the error when applying the patch for database server):
[root@zerosita01 patch_19.3.2.0.0.191119]# cd .. [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# exit [screen is terminating] [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# rpm -qa |grep screen screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# rpm -e screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita01 exa_stack]#
4 – Patch Database Servers
For Database server, I always prefer to patch the node 1 first, and after the other nodes. To do that, we copy the files to another node that has the ssh key for node 1 (if no, you can use another machine or configure now). You can still do this in one external machine if you want.
Before starting the patch is important to show some details. First, the patch is applied by database server patch orchestrator (basically it is a solo patchmgr). The second, it is for database server we need to inform the target version (the version that we want to patch), this is the Exadata Channel version and can be found in the link description of the patch (marked in red):
From Node 2 to Node 1
Preparing the other node
We need to prepare the other node to patch the node 1. We copy the patches, unzip it, install screen (and start the session), and stop all NFS (if mounted).
[root@zerosita02 ~]# cd /radump/ [root@zerosita02 radump]# [root@zerosita02 radump]# mkdir exa_stack [root@zerosita02 radump]# [root@zerosita02 radump]# cd exa_stack/ [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/patchmgr/p21634633_193100_Linux-x86-64.zip ./ [root@zerosita02 exa_stack]# cp /zfs/ZDLRA_PATCHING/19.2.1.1.1/19.2.1.1.1-201910-30614042/EXA-STACK/dbnode/p30439212_193000_Linux-x86-64.zip ./ [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# unzip p21634633_193100_Linux-x86-64.zip Archive: p21634633_193100_Linux-x86-64.zip creating: dbserver_patch_19.191113/ inflating: dbserver_patch_19.191113/README.txt inflating: dbserver_patch_19.191113/md5sum_files.lst inflating: dbserver_patch_19.191113/patchReport.py inflating: dbserver_patch_19.191113/cellboot_usb_pci_path inflating: dbserver_patch_19.191113/dcli inflating: dbserver_patch_19.191113/exadata.img.env extracting: dbserver_patch_19.191113/dbnodeupdate.zip creating: dbserver_patch_19.191113/linux.db.rpms/ inflating: dbserver_patch_19.191113/exadata.img.hw inflating: dbserver_patch_19.191113/patchmgr inflating: dbserver_patch_19.191113/ExadataSendNotification.pm inflating: dbserver_patch_19.191113/ExadataImageNotification.pl inflating: dbserver_patch_19.191113/patchmgr_functions inflating: dbserver_patch_19.191113/imageLogger inflating: dbserver_patch_19.191113/ExaXMLNode.pm [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# ls -l total 1856868 drwxrwxr-x 3 root root 4096 Mar 5 12:30 dbserver_patch_19.191113 -rw-r--r-- 1 root root 438739135 Mar 5 12:20 p21634633_193100_Linux-x86-64.zip -rw-r--r-- 1 root root 1460815642 Mar 5 12:20 p30439212_193000_Linux-x86-64.zip [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# rpm -Uvh /zfs/ZDLRA_PATCHING/19.2.1.1.1/screen-4.1.0-0.25.20120314git3c2946.el7.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:screen-4.1.0-0.25.20120314git3c29################################# [100%] [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# screen -L -RR NODE1-From-Node2-Upgrade [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# umount /zfs [root@zerosita02 exa_stack]#
Precheck
So, from node 2 we check if node 1 is ok with the prechecks (the zip, is the patch that was downloaded, and the target version is the desired for the chosen patch):
[root@zerosita02 dbserver_patch_19.191113]# ./patchmgr -dbnodes /root/db_node01 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -precheck ************************************************************************************************************ NOTE patchmgr release: 19.191113 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2020-03-05 12:41:59 +0100 :Working: Verify SSH equivalence for the root user to zeroadm01 2020-03-05 12:41:59 +0100 :SUCCESS: Verify SSH equivalence for the root user to zeroadm01 2020-03-05 12:42:00 +0100 :Working: Initiate precheck on 1 node(s) 2020-03-05 12:45:34 +0100 :Working: Check free space on zeroadm01 2020-03-05 12:45:37 +0100 :SUCCESS: Check free space on zeroadm01 2020-03-05 12:46:07 +0100 :Working: dbnodeupdate.sh running a precheck on node(s). 2020-03-05 12:47:49 +0100 :SUCCESS: Initiate precheck on node(s). 2020-03-05 12:47:49 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/db_node01 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -precheck 2020-03-05 12:47:50 +0100 :INFO : Precheck attempted on nodes in file /root/db_node01: [zeroadm01] 2020-03-05 12:47:50 +0100 :INFO : Current image version on dbnode(s) is: 2020-03-05 12:47:50 +0100 :INFO : zeroadm01: 19.2.3.0.0.190621 2020-03-05 12:47:50 +0100 :INFO : For details, check the following files in /radump/exa_stack/dbserver_patch_19.191113: 2020-03-05 12:47:50 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2020-03-05 12:47:50 +0100 :INFO : - patchmgr.log 2020-03-05 12:47:50 +0100 :INFO : - patchmgr.trc 2020-03-05 12:47:50 +0100 :INFO : Exit status:0 2020-03-05 12:47:50 +0100 :INFO : Exiting. [root@zerosita02 dbserver_patch_19.191113]#
upgrade
If it is ok with the prereqs, we patch node 1 (from node 2):
[root@zerosita02 dbserver_patch_19.191113]# ./patchmgr -dbnodes /root/db_node01 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -upgrade ************************************************************************************************************ NOTE patchmgr release: 19.191113 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE NOTE Database nodes will reboot during the update process. NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2020-03-05 12:52:12 +0100 :Working: Verify SSH equivalence for the root user to zeroadm01 2020-03-05 12:52:13 +0100 :SUCCESS: Verify SSH equivalence for the root user to zeroadm01 2020-03-05 12:52:14 +0100 :Working: Initiate prepare steps on node(s). 2020-03-05 12:52:14 +0100 :Working: Check free space on zeroadm01 2020-03-05 12:52:18 +0100 :SUCCESS: Check free space on zeroadm01 2020-03-05 12:52:50 +0100 :SUCCESS: Initiate prepare steps on node(s). 2020-03-05 12:52:50 +0100 :Working: Initiate update on 1 node(s). 2020-03-05 12:52:50 +0100 :Working: dbnodeupdate.sh running a backup on 1 node(s). 2020-03-05 12:57:34 +0100 :SUCCESS: dbnodeupdate.sh running a backup on 1 node(s). 2020-03-05 12:57:34 +0100 :Working: Initiate update on node(s) 2020-03-05 12:57:35 +0100 :Working: Get information about any required OS upgrades from node(s). 2020-03-05 12:57:45 +0100 :SUCCESS: Get information about any required OS upgrades from node(s). 2020-03-05 12:57:45 +0100 :Working: dbnodeupdate.sh running an update step on all nodes. 2020-03-05 13:12:17 +0100 :INFO : zeroadm01 is ready to reboot. 2020-03-05 13:12:17 +0100 :SUCCESS: dbnodeupdate.sh running an update step on all nodes. 2020-03-05 13:12:26 +0100 :Working: Initiate reboot on node(s) 2020-03-05 13:12:28 +0100 :SUCCESS: Initiate reboot on node(s) 2020-03-05 13:12:28 +0100 :Working: Waiting to ensure zeroadm01 is down before reboot. 2020-03-05 13:12:52 +0100 :SUCCESS: Waiting to ensure zeroadm01 is down before reboot. 2020-03-05 13:12:52 +0100 :Working: Waiting to ensure zeroadm01 is up after reboot. 2020-03-05 13:15:46 +0100 :SUCCESS: Waiting to ensure zeroadm01 is up after reboot. 2020-03-05 13:15:46 +0100 :Working: Waiting to connect to zeroadm01 with SSH. During Linux upgrades this can take some time. 2020-03-05 13:35:10 +0100 :SUCCESS: Waiting to connect to zeroadm01 with SSH. During Linux upgrades this can take some time. 2020-03-05 13:35:10 +0100 :Working: Wait for zeroadm01 is ready for the completion step of update. 2020-03-05 13:35:10 +0100 :SUCCESS: Wait for zeroadm01 is ready for the completion step of update. 2020-03-05 13:35:10 +0100 :Working: Initiate completion step from dbnodeupdate.sh on node(s) 2020-03-05 13:42:37 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on zeroadm01 2020-03-05 13:42:50 +0100 :SUCCESS: Initiate update on node(s). 2020-03-05 13:42:51 +0100 :SUCCESS: Initiate update on 0 node(s). [INFO ] Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_upgrade_050320125211.tbz -rw-r--r-- 1 root root 2397370 Mar 5 13:42 Diag_patchmgr_dbnode_upgrade_050320125211.tbz 2020-03-05 13:42:52 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/db_node01 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -upgrade 2020-03-05 13:42:52 +0100 :INFO : Upgrade attempted on nodes in file /root/db_node01: [zeroadm01] 2020-03-05 13:42:52 +0100 :INFO : Current image version on dbnode(s) is: 2020-03-05 13:42:52 +0100 :INFO : zeroadm01: 19.3.2.0.0.191119 2020-03-05 13:42:52 +0100 :INFO : For details, check the following files in /radump/exa_stack/dbserver_patch_19.191113: 2020-03-05 13:42:52 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2020-03-05 13:42:52 +0100 :INFO : - patchmgr.log 2020-03-05 13:42:52 +0100 :INFO : - patchmgr.trc 2020-03-05 13:42:52 +0100 :INFO : Exit status:0 2020-03-05 13:42:52 +0100 :INFO : Exiting. You have new mail in /var/spool/mail/root [root@zerosita02 dbserver_patch_19.191113]#
Post-patch of Node 1
If the patch was Ok, we can finish the patch at node 2. This means exist screen session and remove it:
[root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# rpm -qa |grep screen screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# [root@zerosita02 exa_stack]# rpm -e screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita02 exa_stack]#
From Node 1 to Node 2 (or others)
Preparing node 1
Since we upgraded node 1 in the above we need to reinstall screen and create the session. Check that maybe you need a new rpm version if you upgraded (as an example) from OEL 6 to OEL 7:
[root@zerosita01 ~]# cd /radump/exa_stack/ [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# rpm -Uvh /zfs/ZDLRA_PATCHING/19.2.1.1.1/screen-4.1.0-0.25.20120314git3c2946.el7.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:screen-4.1.0-0.25.20120314git3c29################################# [100%] [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# umount /zfs/ [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# screen -L -RR NODE2-From-Node1-Upgrade [root@zerosita01 exa_stack]#
Precheck
We need to precheck the other nodes to check inconsistencies. In this scenario, the file has the list for other nodes (just one):
[root@zerosita01 dbserver_patch_19.191113]# ./patchmgr -dbnodes /root/db_group_zeroadm02 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -precheck ************************************************************************************************************ NOTE patchmgr release: 19.191113 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2020-03-05 13:57:04 +0100 :Working: Verify SSH equivalence for the root user to zeroadm02 2020-03-05 13:57:05 +0100 :SUCCESS: Verify SSH equivalence for the root user to zeroadm02 2020-03-05 13:57:06 +0100 :Working: Initiate precheck on 1 node(s) 2020-03-05 14:00:39 +0100 :Working: Check free space on zeroadm02 2020-03-05 14:00:42 +0100 :SUCCESS: Check free space on zeroadm02 2020-03-05 14:01:12 +0100 :Working: dbnodeupdate.sh running a precheck on node(s). 2020-03-05 14:02:54 +0100 :SUCCESS: Initiate precheck on node(s). 2020-03-05 14:02:54 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/db_group_zeroadm02 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -precheck 2020-03-05 14:02:55 +0100 :INFO : Precheck attempted on nodes in file /root/db_group_zeroadm02: [zeroadm02] 2020-03-05 14:02:55 +0100 :INFO : Current image version on dbnode(s) is: 2020-03-05 14:02:55 +0100 :INFO : zeroadm02: 19.2.3.0.0.190621 2020-03-05 14:02:55 +0100 :INFO : For details, check the following files in /radump/exa_stack/dbserver_patch_19.191113: 2020-03-05 14:02:55 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2020-03-05 14:02:55 +0100 :INFO : - patchmgr.log 2020-03-05 14:02:55 +0100 :INFO : - patchmgr.trc 2020-03-05 14:02:55 +0100 :INFO : Exit status:0 2020-03-05 14:02:55 +0100 :INFO : Exiting. You have new mail in /var/spool/mail/root [root@zerosita01 dbserver_patch_19.191113]#
upgrade
If it is OK, we can do the upgrade:
[root@zerosita01 dbserver_patch_19.191113]# ./patchmgr -dbnodes /root/db_group_zeroadm02 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -upgrade ************************************************************************************************************ NOTE patchmgr release: 19.191113 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip) NOTE NOTE Database nodes will reboot during the update process. NOTE WARNING Do not interrupt the patchmgr session. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot database nodes during update or rollback. WARNING Do not open logfiles in write mode and do not try to alter them. ************************************************************************************************************ 2020-03-05 14:14:24 +0100 :Working: Verify SSH equivalence for the root user to zeroadm02 2020-03-05 14:14:25 +0100 :SUCCESS: Verify SSH equivalence for the root user to zeroadm02 2020-03-05 14:14:26 +0100 :Working: Initiate prepare steps on node(s). 2020-03-05 14:14:26 +0100 :Working: Check free space on zeroadm02 2020-03-05 14:14:30 +0100 :SUCCESS: Check free space on zeroadm02 2020-03-05 14:15:02 +0100 :SUCCESS: Initiate prepare steps on node(s). 2020-03-05 14:15:02 +0100 :Working: Initiate update on 1 node(s). 2020-03-05 14:15:02 +0100 :Working: dbnodeupdate.sh running a backup on 1 node(s). 2020-03-05 14:19:56 +0100 :SUCCESS: dbnodeupdate.sh running a backup on 1 node(s). 2020-03-05 14:19:56 +0100 :Working: Initiate update on node(s) 2020-03-05 14:19:56 +0100 :Working: Get information about any required OS upgrades from node(s). 2020-03-05 14:20:06 +0100 :SUCCESS: Get information about any required OS upgrades from node(s). 2020-03-05 14:20:06 +0100 :Working: dbnodeupdate.sh running an update step on all nodes. 2020-03-05 14:34:37 +0100 :INFO : zeroadm02 is ready to reboot. 2020-03-05 14:34:37 +0100 :SUCCESS: dbnodeupdate.sh running an update step on all nodes. 2020-03-05 14:34:44 +0100 :Working: Initiate reboot on node(s) 2020-03-05 14:34:47 +0100 :SUCCESS: Initiate reboot on node(s) 2020-03-05 14:34:47 +0100 :Working: Waiting to ensure zeroadm02 is down before reboot. 2020-03-05 14:35:11 +0100 :SUCCESS: Waiting to ensure zeroadm02 is down before reboot. 2020-03-05 14:35:11 +0100 :Working: Waiting to ensure zeroadm02 is up after reboot. 2020-03-05 14:38:04 +0100 :SUCCESS: Waiting to ensure zeroadm02 is up after reboot. 2020-03-05 14:38:04 +0100 :Working: Waiting to connect to zeroadm02 with SSH. During Linux upgrades this can take some time. 2020-03-05 15:05:13 +0100 :SUCCESS: Waiting to connect to zeroadm02 with SSH. During Linux upgrades this can take some time. 2020-03-05 15:05:13 +0100 :Working: Wait for zeroadm02 is ready for the completion step of update. 2020-03-05 15:05:13 +0100 :SUCCESS: Wait for zeroadm02 is ready for the completion step of update. 2020-03-05 15:05:14 +0100 :Working: Initiate completion step from dbnodeupdate.sh on node(s) 2020-03-05 15:12:39 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on zeroadm02 2020-03-05 15:12:53 +0100 :SUCCESS: Initiate update on node(s). 2020-03-05 15:12:53 +0100 :SUCCESS: Initiate update on 0 node(s). [INFO ] Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_upgrade_050320141423.tbz -rw-r--r-- 1 root root 2351811 Mar 5 15:12 Diag_patchmgr_dbnode_upgrade_050320141423.tbz 2020-03-05 15:12:54 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/db_group_zeroadm02 -iso_repo /radump/exa_stack/p30439212_193000_Linux-x86-64.zip -target_version 19.3.2.0.0.191119 -upgrade 2020-03-05 15:12:54 +0100 :INFO : Upgrade attempted on nodes in file /root/db_group_zeroadm02: [zeroadm02] 2020-03-05 15:12:54 +0100 :INFO : Current image version on dbnode(s) is: 2020-03-05 15:12:54 +0100 :INFO : zeroadm02: 19.3.2.0.0.191119 2020-03-05 15:12:54 +0100 :INFO : For details, check the following files in /radump/exa_stack/dbserver_patch_19.191113: 2020-03-05 15:12:54 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log 2020-03-05 15:12:54 +0100 :INFO : - patchmgr.log 2020-03-05 15:12:54 +0100 :INFO : - patchmgr.trc 2020-03-05 15:12:54 +0100 :INFO : Exit status:0 2020-03-05 15:12:54 +0100 :INFO : Exiting. You have new mail in /var/spool/mail/root [root@zerosita01 dbserver_patch_19.191113]#
Post-patch of other nodes
After success, we can exit the session and remove the screen:
[root@zerosita01 dbserver_patch_19.191113]# exit [screen is terminating] [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# rpm -qa |grep screen screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# rpm -e screen-4.1.0-0.25.20120314git3c2946.el7.x86_64 [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]#
5 – Post-Patch
The post-patch is simple, we can just check the versions that we are running. If the upgrade was ok for all servers and switched (if the version is the desired one):
[root@zerosita01 exa_stack]# dcli -l root -g /root/cell_group "imageinfo" |grep "Active image version" zerocell01: Active image version: 19.3.2.0.0.191119 zerocell02: Active image version: 19.3.2.0.0.191119 zerocell03: Active image version: 19.3.2.0.0.191119 zerocell04: Active image version: 19.3.2.0.0.191119 zerocell05: Active image version: 19.3.2.0.0.191119 zerocell06: Active image version: 19.3.2.0.0.191119 [root@zerosita01 exa_stack]# dcli -l root -g /root/db_group "imageinfo" |grep "Active image version" [root@zerosita01 exa_stack]# dcli -l root -g /root/db_group "imageinfo" |grep "Image version" zerosita01: Image version: 19.3.2.0.0.191119 zerosita02: Image version: 19.3.2.0.0.191119 [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# dcli -l root -g /root/ib_group "version" |grep "36p" zdls-iba01: SUN DCS 36p version: 2.2.14-1 zdls-ibb01: SUN DCS 36p version: 2.2.14-1 [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# dcli -l root -g /root/cell_group "ipmitool sunoem led get" |grep ": SERVICE" zerocell01: SERVICE | OFF zerocell02: SERVICE | OFF zerocell03: SERVICE | OFF zerocell04: SERVICE | OFF zerocell05: SERVICE | OFF zerocell06: SERVICE | OFF [root@zerosita01 exa_stack]# dcli -l root -g /root/db_group "ipmitool sunoem led get" |grep ": SERVICE" zerosita01: SERVICE | OFF zerosita02: SERVICE | OFF [root@zerosita01 exa_stack]#
6 – Restart CRS and Cluster
Since at step 1 we stopped the cluster, we can now start CRS and GI:
[root@zerosita01 exa_stack]# /u01/app/19.0.0.0/grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# ssh -q zerosita02 Last login: Thu Mar 5 15:17:03 CET 2020 from 99.234.123.138 on ssh Last login: Thu Mar 5 15:17:10 2020 from zerosita01.zero.flisk.net [root@zerosita02 ~]# [root@zerosita02 ~]# /u01/app/19.0.0.0/grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# logout [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# /u01/app/19.0.0.0/grid/bin/crsctl start cluster -all [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# /u01/app/19.0.0.0/grid/bin/crsctl start cluster -all CRS-4690: Oracle Clusterware is already running on 'zerosita01' CRS-4690: Oracle Clusterware is already running on 'zerosita02' CRS-4000: Command Start failed, or completed with errors. [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]#
Finishing Patch Apply
After we check that everything is fine, all the patches were applied (at Storage, Database server, and InfiniBand switches), and there is no HW error (or alerts at storage side), we can finish the patch.
For ZDLRA
If you are patching ZDLRA we need to check the OS system in this case. Since ZDLRA needs some special parameters and the OS was upgraded to the vanilla Exadata version, we need to fix it.
This is done using the script /opt/oracle.RecoveryAppliance/os/setup_os.sh executed in both nodes:
[root@zerosita01 exa_stack]# /opt/oracle.RecoveryAppliance/os/setup_os.sh cat: /opt/oracle.SupportTools/onecommand/tape.xml: No such file or directory Enabled rpcbind Started rpcbind Skipping Qlogic setup. No tape selection made. Skipping LVRADump LVM Creation. It already exists /dev/mapper/VGExaDb-LVRADump: Linux rev 1.0 ext3 filesystem data, UUID=844a0941-ab23-4524-86a7-493c466b50fa, volume name "LVRADump" (needs journal recovery) (large files) Skipping LVRADump Filesystem Creation. It already exists Skipping /radump mountpoint creation, it already exists Skipping fstab update. LVRADump exists. /dev/mapper/VGExaDb-LVRADump on /radump type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRADump mount, it already exists. Setting oracle permissions on /radump Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRADump Skipping LVRABackup LVM Creation. It already exists /dev/mapper/VGExaDb-LVRABackup: Linux rev 1.0 ext3 filesystem data, UUID=36b6cee7-abb6-41c7-80bc-a4080c9180a6, volume name "LVRABackup" (needs journal recovery) (large files) Skipping LVRABackup Filesystem Creation. It already exists Skipping /rabackup mountpoint creation, it already exists Skipping fstab update. LVRABackup exists. /dev/mapper/VGExaDb-LVRABackup on /rabackup type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRABackup mount, it already exists. Setting oracle permissions on /rabackup Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRABackup Skipping LVOBIndex LVM Creation. It already exists /dev/mapper/VGExaDb-LVOBIndex: Linux rev 1.0 ext3 filesystem data, UUID=90447c5b-0bf1-4639-bfcd-1144dc1a57e4, volume name "LVOBIndex" (needs journal recovery) (large files) Skipping LVOBIndex Filesystem Creation. It already exists Skipping /obindex mountpoint creation, it already exists Skipping fstab update. LVOBIndex exists. /dev/mapper/VGExaDb-LVOBIndex on /obindex type ext3 (rw,nodev,relatime,data=ordered) Skipping LVOBIndex mount, it already exists. Setting oracle permissions on /obindex Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVOBIndex Skipping LVRANfs LVM Creation. It already exists /dev/mapper/VGExaDb-LVRANfs: Linux rev 1.0 ext3 filesystem data, UUID=90bf4fa3-385a-4b58-923c-c0a5c6651247, volume name "LVRANfs" (needs journal recovery) (large files) Skipping LVRANfs Filesystem Creation. It already exists Skipping /ranfs mountpoint creation, it already exists Skipping fstab update. LVRANfs exists. /dev/mapper/VGExaDb-LVRANfs on /ranfs type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRANfs mount, it already exists. Setting oracle permissions on /ranfs Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRANfs Skipping adding fuse group to oracle. It exists. Skipping updates to fuse.conf. It exists. Skipping mkdir /dbfs_obdbfs. It exists. Skipping mkdir /dbfs_repdbfs. It exists. Skipping mkdir /u01/okv. It exists. Skipping mkdir /ranfs. It exists. Moved existing /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libnnz19.so link to /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libnnz19.so.old Moved existing /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libclntsh.so.19.1 link to /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libclntsh.so.19.1.old Moved existing /lib64/libfuse.so.2 link to /lib64/libfuse.so.2.old Found user raext. Skipping Found user railm. Skipping Successfully modified oracle user Skipping. denied ssh access already. Wrote backup of BIOS configuration to '/opt/oracle.RecoveryAppliance/config/bios.backup.xml'. Hyper-threading is disabled. Nothing to do. No bondeth1 interface. Nothing to change. Skipping mkdir /u01/app/emagent. It exists. Found uncompress. Skipping link oracle crontab successfully installed. /etc/cron.allow successfully installed. Updated perms on diag dir Skipping /u01/app/19.0.0.0/grid/crs/script/mount-ob_dbfs.sh, not installed. Skipping /u01/app/19.0.0.0/grid/crs/script/mount-rep_dbfs.sh, not installed. update_dbfs_script.pl - completed. ZDLRA Banner already enabled, Nothing to do. expanding /u01... Space available on /u01: 630.37 Units: GiB Available space GiB: 630 Checking the volume logical configuration... Current configuration: LSize 300.00g current /u01 size: 296 /u01 already expanded... Skipping root path set - file exists... Skipping Fixing perms on /raacfs Fixed perms on /raacfs Fixing perms on /osbcat Fixed perms on /osbcat Successfully setup OS for Recovery Appliance [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# ssh -q zerosita02 Last login: Thu Mar 5 15:23:49 CET 2020 from 99.234.123.138 on ssh Last login: Thu Mar 5 15:24:40 2020 from zerosita01.zero.flisk.net [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# /opt/oracle.RecoveryAppliance/os/setup_os.sh cat: /opt/oracle.SupportTools/onecommand/tape.xml: No such file or directory Enabled rpcbind Started rpcbind Skipping Qlogic setup. No tape selection made. Skipping LVRADump LVM Creation. It already exists /dev/mapper/VGExaDb-LVRADump: Linux rev 1.0 ext3 filesystem data, UUID=0868744a-fdae-46ab-9629-c7102bdae0b8, volume name "LVRADump" (needs journal recovery) (large files) Skipping LVRADump Filesystem Creation. It already exists Skipping /radump mountpoint creation, it already exists Skipping fstab update. LVRADump exists. /dev/mapper/VGExaDb-LVRADump on /radump type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRADump mount, it already exists. Setting oracle permissions on /radump Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRADump Skipping LVRABackup LVM Creation. It already exists /dev/mapper/VGExaDb-LVRABackup: Linux rev 1.0 ext3 filesystem data, UUID=c4bdfce5-522f-4597-bd37-0298b0cfc052, volume name "LVRABackup" (needs journal recovery) (large files) Skipping LVRABackup Filesystem Creation. It already exists Skipping /rabackup mountpoint creation, it already exists Skipping fstab update. LVRABackup exists. /dev/mapper/VGExaDb-LVRABackup on /rabackup type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRABackup mount, it already exists. Setting oracle permissions on /rabackup Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRABackup Skipping LVOBIndex LVM Creation. It already exists /dev/mapper/VGExaDb-LVOBIndex: Linux rev 1.0 ext3 filesystem data, UUID=5dcf5c69-ded6-4fc5-a5f0-3a9f3435681c, volume name "LVOBIndex" (needs journal recovery) (large files) Skipping LVOBIndex Filesystem Creation. It already exists Skipping /obindex mountpoint creation, it already exists Skipping fstab update. LVOBIndex exists. /dev/mapper/VGExaDb-LVOBIndex on /obindex type ext3 (rw,nodev,relatime,data=ordered) Skipping LVOBIndex mount, it already exists. Setting oracle permissions on /obindex Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVOBIndex Skipping LVRANfs LVM Creation. It already exists /dev/mapper/VGExaDb-LVRANfs: Linux rev 1.0 ext3 filesystem data, UUID=2e2917e3-8c7f-4b23-b46d-247e1a26b2bd, volume name "LVRANfs" (needs journal recovery) (large files) Skipping LVRANfs Filesystem Creation. It already exists Skipping /ranfs mountpoint creation, it already exists Skipping fstab update. LVRANfs exists. /dev/mapper/VGExaDb-LVRANfs on /ranfs type ext3 (rw,nodev,relatime,data=ordered) Skipping LVRANfs mount, it already exists. Setting oracle permissions on /ranfs Permissions for oracle Set tune2fs 1.42.9 (28-Dec-2013) Setting maximal mount count to -1 Setting interval between checks to 0 seconds Disabled automatic tune2fs checking of LVRANfs Skipping adding fuse group to oracle. It exists. Skipping updates to fuse.conf. It exists. Skipping mkdir /dbfs_obdbfs. It exists. Skipping mkdir /dbfs_repdbfs. It exists. Skipping mkdir /u01/okv. It exists. Skipping mkdir /ranfs. It exists. Moved existing /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libnnz19.so link to /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libnnz19.so.old Moved existing /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libclntsh.so.19.1 link to /u01/app/oracle/product/19.0.0.0/dbhome_1/lib/libclntsh.so.19.1.old Moved existing /lib64/libfuse.so.2 link to /lib64/libfuse.so.2.old Found user raext. Skipping Found user railm. Skipping Successfully modified oracle user Skipping. denied ssh access already. Wrote backup of BIOS configuration to '/opt/oracle.RecoveryAppliance/config/bios.backup.xml'. Hyper-threading is disabled. Nothing to do. No bondeth1 interface. Nothing to change. Skipping mkdir /u01/app/emagent. It exists. Found uncompress. Skipping link oracle crontab successfully installed. /etc/cron.allow successfully installed. Updated perms on diag dir Skipping /u01/app/19.0.0.0/grid/crs/script/mount-ob_dbfs.sh, not installed. Skipping /u01/app/19.0.0.0/grid/crs/script/mount-rep_dbfs.sh, not installed. update_dbfs_script.pl - completed. ZDLRA Banner already enabled, Nothing to do. expanding /u01... Space available on /u01: 630.37 Units: GiB Available space GiB: 630 Checking the volume logical configuration... Current configuration: LSize 300.00g current /u01 size: 296 /u01 already expanded... Skipping root path set - file exists... Skipping Fixing perms on /raacfs Fixed perms on /raacfs Fixing perms on /osbcat Fixed perms on /osbcat Successfully setup OS for Recovery Appliance [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# [root@zerosita02 ~]# exit logout [root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]#
Startup ZDLRA
And after that we can startup the ZDLRA database (and RA software) that we stopped in the beginning:
[root@zerosita01 exa_stack]# [root@zerosita01 exa_stack]# su - oracle Last login: Thu Mar 5 15:26:22 CET 2020 from 99.234.123.138 on ssh [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ srvctl start database -d zdlras [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$ sqlplus rasys/ra SQL*Plus: Release 19.0.0.0.0 - Production on Thu Mar 5 15:28:00 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Mar 05 2020 11:24:37 +01:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> SELECT state FROM ra_server; STATE ------------- OFF SQL> exec dbms_ra.startup; PL/SQL procedure successfully completed. SQL> SELECT state FROM ra_server; STATE ------------- ON SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 [oracle@zerosita01 ~]$ [oracle@zerosita01 ~]$
Conclusion
Patching Oracle Engineering System is not complicated, but some details need to be checked to do correctly. The way that we patch (Rolling/Non-Rolling), files with the hostname, the ssh between servers, version, and files used are important. Several details that were showed above.
For ZDLRA it is more rigid since we need to take care of the ZDLRA database, RA software, and operating system parameters.
It is impossible to describe all the details in one post, maybe one post for every step (like the file with the hostnames of servers that we will apply the patch and pass as parameter of patchmgr, or some details about how to patch in rolling mode). But I tried to cover all the points here (and in my previous post about the same topic). With this post, I think that you can have one idea and use it as a guide to patching the Exadata stack. The steps are in order and if you have doubts, comment here.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Good stuff thanks, from what I’ve read even as being simple DBA I feel I can patch ZDLRA by myself :).
Regards.
G
Pingback: ZDLRA, Patch/Update the Recovery Appliance | Fernando Simon
Awesome. Crystal clear Explanation.
Pingback: ZDLRA patching | Amardeep Sidhu