On 08/March/2023 the Oracle Exadata team released version 23.1.0.0.0 and this include a significant change, OEL 8. I already explained that in my first post that you can read here. Here I will show how to patch to the 23.1.0.0 version for switch and storage cells.
Patching
As you know, I am working with Exadata since 2010 and have already posted about how to upgrade to the 19x version, 18.x version, version 12.x (Portuguese only), and many other details for Oracle Engineered Systems. Fortunately, I had the opportunity to apply the 23.1 version over one environment and will show the details.
Here, my environment is:
- Exadata X6 for storage and dbnodes.
- InfiniBand Switches.
- Virtualized configuration (dom0 and domU).
- dom0 running over version 22.1.0.9.
- domU running over version 22.1.0.9.
- Grid Infrastructure running version 19.17.
Since I am running with dom0/domU, my base machine (from where I will call most of the patches) is the dom0. There, I have ssh passwordless/keyless to all other cells, dbnodes, domU, and switches.
Before you start the patch please check the readme for the patch and identify if you have everything in compliance. Do not start any patch if you meet the requirements. Even from a simple database version, grid, and switch versions. And as well, do not start the patch if your machine has HW errors. So, please read the note Exadata System Software 23.1.0.0.0 Update (32829291) (Doc ID 2772585.1).
Patching Switch
I prefer to patch starting from the switch because, just in case, if the new version for storage software needs something special at interconnect network level, everything is already there and the switch can send the packages correctly.
The first is to have one file (here I have at /root) with the name of all switches hostnames that you have. This helps to call dcli and patch commands:
[DOM0 - root@exadom0m01 ~]$ cat /roort/ibs_group exasw-iba01 exasw-ibb01 [DOM0 - root@exadom0m01 ~]$
So, I downloaded patch 32829304 in the folder and unzip it:
[DOM0 - root@exadom0m01 ~]$ cd /EXAVMIMAGES/patches/switch/ [DOM0 - root@exadom0m01 switch]$ [DOM0 - root@exadom0m01 switch]$ ls -l total 3979264 -rw-r--r-- 1 root root 4074685740 Mar 9 09:09 p32829304_231000_Linux-x86-64.zip [DOM0 - root@exadom0m01 switch]$ [DOM0 - root@exadom0m01 switch]$ unzip -q p32829304_231000_Linux-x86-64.zip [DOM0 - root@exadom0m01 switch]$ [DOM0 - root@exadom0m01 switch]$ ls -l total 3979268 -rw-r--r-- 1 root root 4074685740 Mar 9 09:09 p32829304_231000_Linux-x86-64.zip drwxrwxr-x 6 root root 3896 Feb 26 00:43 patch_switch_23.1.0.0.0.230225.1 [DOM0 - root@exadom0m01 switch]$
After that, I can start my prechecks. First, I check my current running version (look that I use dcli and the file with my switches):
[DOM0 - root@exadom0m01 switch]$ dcli -l root -g /root/ibs_group "version" exasw-iba01: SUN DCS 36p version: 2.2.16-1 exasw-iba01: Build time: Sep 3 2020 08:23:26 exasw-iba01: SP board info: exasw-iba01: Manufacturing Date: N/A exasw-iba01: Serial Number: "NED640259" exasw-iba01: Hardware Revision: 0x0100 exasw-iba01: Firmware Revision: 0x0000 exasw-iba01: BIOS version: NUP1R918 exasw-iba01: BIOS date: 01/19/2016 exasw-ibb01: SUN DCS 36p version: 2.2.16-1 exasw-ibb01: Build time: Sep 3 2020 08:23:26 exasw-ibb01: SP board info: exasw-ibb01: Manufacturing Date: N/A exasw-ibb01: Serial Number: "NED510150" exasw-ibb01: Hardware Revision: 0x0100 exasw-ibb01: Firmware Revision: 0x0000 exasw-ibb01: BIOS version: NUP1R918 exasw-ibb01: BIOS date: 01/19/2016 [DOM0 - root@exadom0m01 switch]$
And we can call the precheck:
[DOM0 - root@exadom0m01 switch]$ ./patch_switch_23.1.0.0.0.230225.1/patchmgr -ibswitches /root/ibs_group -upgrade -ibswitch_precheck -rolling
./patch_switch_23.1.0.0.0.230225.1/patchmgr: line 3117: /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/timing_record.sh: No such file or directory
2023-03-09 10:48:53 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2023-03-09 10:48:55 +0100 :INFO : SSH equivalency verified to host exasw-iba01
2023-03-09 10:48:56 +0100 :INFO : SSH equivalency verified to host exasw-ibb01
./patch_switch_23.1.0.0.0.230225.1/patchmgr: line 3213: /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/timing_record.sh: No such file or directory
2023-03-09 10:48:56 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2023-03-09 10:48:58 +0100 1 of 1 :Working: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started 2023-03-09 10:48:58 +0100 -----
[NOTE ] Log file at /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exasw-iba01 exasw-ibb01 )
[SUCCESS ] Verifying Network connectivity to exasw-iba01
[SUCCESS ] Verifying Network connectivity to exasw-ibb01
[SUCCESS ] Validating verify-topology output
[SUCCESS ] SubnetManager Partition validation on IB Switches
[INFO ] Master Subnet Manager is set to "exasw-ibb01" in all Switches
|||||
[INFO ] ---------- Starting with InfiniBand Switch exasw-ibb01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.16-5 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host exadom0m01.simon.mynet.lu from the InfiniBand Switch exasw-ibb01.
[INFO ] Starting pre-update validation on exasw-ibb01
[SUCCESS ] Verifying that /tmp has 150M in exasw-ibb01, found 492M
[SUCCESS ] Verifying that / has 20M in exasw-ibb01, found 28M
[SUCCESS ] NTP daemon is running on exasw-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 10:49:15
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-ibb01
[SUCCESS ] Verifying that the patchmgr host exadom0m01.simon.mynet.lu is recognized on the InfiniBand Switch exasw-ibb01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on exasw-ibb01
[INFO ] Finished pre-update validation on exasw-ibb01
[SUCCESS ] Pre-update validation on exasw-ibb01
[SUCCESS ] Prereq check on exasw-ibb01
|||||
[INFO ] ---------- Starting with InfiniBand Switch exasw-iba01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.16-5 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host exadom0m01.simon.mynet.lu from the InfiniBand Switch exasw-iba01.
[INFO ] Starting pre-update validation on exasw-iba01
[SUCCESS ] Verifying that /tmp has 150M in exasw-iba01, found 492M
[SUCCESS ] Verifying that / has 20M in exasw-iba01, found 28M
[SUCCESS ] NTP daemon is running on exasw-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 10:49:34
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-iba01
[SUCCESS ] Verifying that the patchmgr host exadom0m01.simon.mynet.lu is recognized on the InfiniBand Switch exasw-iba01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on exasw-iba01
[INFO ] Finished pre-update validation on exasw-iba01
[SUCCESS ] Pre-update validation on exasw-iba01
[SUCCESS ] Prereq check on exasw-iba01
[SUCCESS ] Overall status
/////
----- InfiniBand switch update process ended 2023-03-09 10:49:47 +0100 -----
2023-03-09 10:49:50 +0100 1 of 1 :SUCCESS: Initiate pre-upgrade validation check on InfiniBand switch(es).
2023-03-09 10:49:50 +0100 :SUCCESS: Completed run of command: ./patch_switch_23.1.0.0.0.230225.1/patchmgr -ibswitches /root/ibs_group -upgrade -ibswitch_precheck -rolling
2023-03-09 10:49:50 +0100 :INFO : upgrade performed on switch(es) in file /root/ibs_group: [exasw-iba01 exasw-ibb01]
2023-03-09 10:49:50 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1:
2023-03-09 10:49:50 +0100 :INFO : - upgradeIBSwitch.log
2023-03-09 10:49:50 +0100 :INFO : - upgradeIBSwitch.trc
2023-03-09 10:49:50 +0100 :INFO : - patchmgr.stdout
2023-03-09 10:49:50 +0100 :INFO : - patchmgr.stderr
2023-03-09 10:49:50 +0100 :INFO : - patchmgr.log
2023-03-09 10:49:50 +0100 :INFO : - patchmgr.trc
2023-03-09 10:49:50 +0100 :INFO : Exit status:0
2023-03-09 10:49:50 +0100 :INFO : Exiting.
[DOM0 - root@exadom0m01 switch]$
As you can see, I am checking in rolling mode, so, will be one switch per time and nothing need to be shutdown in the system. Before the patch I start screen to guarantee that in case of network failure, my session continues to run:
[DOM0 - root@exadom0m01 switch]$ screen -L -RR Patch-Switch [DOM0 - root@exadom0m01 switch]$
And after we can call the patch, from version 2.2.16-1 to 2.2.16_6. Look that I specify nothing (like version), I just defined that will be the “–upgrade” and in “–rolling” mode:
[DOM0 - root@exadom0m01 switch]$ ./patch_switch_23.1.0.0.0.230225.1/patchmgr -ibswitches /root/ibs_group -upgrade -rolling
./patch_switch_23.1.0.0.0.230225.1/patchmgr: line 3117: /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/timing_record.sh: No such file or directory
2023-03-09 10:51:40 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2023-03-09 10:51:42 +0100 :INFO : SSH equivalency verified to host exasw-iba01
2023-03-09 10:51:43 +0100 :INFO : SSH equivalency verified to host exasw-ibb01
./patch_switch_23.1.0.0.0.230225.1/patchmgr: line 3213: /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/timing_record.sh: No such file or directory
2023-03-09 10:51:43 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2023-03-09 10:51:45 +0100 1 of 1 :Working: Initiate upgrade of InfiniBand switches to 2.2.16-6. Expect up to 40 minutes for each switch
----- InfiniBand switch update process started 2023-03-09 10:51:45 +0100 -----
[NOTE ] Log file at /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( exasw-iba01 exasw-ibb01 )
[SUCCESS ] Verifying Network connectivity to exasw-iba01
[SUCCESS ] Verifying Network connectivity to exasw-ibb01
[SUCCESS ] Validating verify-topology output
[SUCCESS ] SubnetManager Partition validation on IB Switches
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.16_6
[INFO ] Master Subnet Manager is set to "exasw-ibb01" in all Switches
-----
[INFO ] ---------- Starting with InfiniBand Switch exasw-ibb01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.16-5 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host exadom0m01.simon.mynet.lu from the InfiniBand Switch exasw-ibb01.
[INFO ] Starting pre-update validation on exasw-ibb01
[SUCCESS ] Verifying that /tmp has 150M in exasw-ibb01, found 492M
[SUCCESS ] Verifying that / has 20M in exasw-ibb01, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch exasw-ibb01
[SUCCESS ] NTP daemon is running on exasw-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 10:52:02
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-ibb01
[SUCCESS ] Verifying that the patchmgr host exadom0m01.simon.mynet.lu is recognized on the InfiniBand Switch exasw-ibb01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on exasw-ibb01
[INFO ] Finished pre-update validation on exasw-ibb01
[SUCCESS ] Pre-update validation on exasw-ibb01
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on exasw-ibb01
[INFO ] Starting upgrade on exasw-ibb01 to 2.2.16_6. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting exasw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS ] Load firmware 2.2.16_6 onto exasw-ibb01
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exasw-ibb01
[INFO ] Set SMPriority to 5 on exasw-ibb01
[INFO ] Starting post-update validation on exasw-ibb01
[SUCCESS ] Service opensmd is running on InfiniBand Switch exasw-ibb01
[SUCCESS ] NTP daemon is running on exasw-ibb01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 11:09:01
[INFO ] /conf/configvalid is 1
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-ibb01
[SUCCESS ] Execute plugin check for Post Patch on exasw-ibb01
[INFO ] Finished post-update validation on exasw-ibb01
[SUCCESS ] Post-update validation on exasw-ibb01
[SUCCESS ] SMPartition information is consistent on IB Switch exasw-ibb01 post upgrade
[SUCCESS ] Update InfiniBand switch exasw-ibb01 to 2.2.16_6
|||||
[INFO ] ---------- Starting with InfiniBand Switch exasw-iba01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.16-5 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host exadom0m01.simon.mynet.lu from the InfiniBand Switch exasw-iba01.
[INFO ] Starting pre-update validation on exasw-iba01
[SUCCESS ] Verifying that /tmp has 150M in exasw-iba01, found 492M
[SUCCESS ] Verifying that / has 20M in exasw-iba01, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch exasw-iba01
[SUCCESS ] NTP daemon is running on exasw-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 11:09:33
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-iba01
[SUCCESS ] Verifying that the patchmgr host exadom0m01.simon.mynet.lu is recognized on the InfiniBand Switch exasw-iba01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on exasw-iba01
[INFO ] Finished pre-update validation on exasw-iba01
[SUCCESS ] Pre-update validation on exasw-iba01
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on exasw-iba01
[INFO ] Starting upgrade on exasw-iba01 to 2.2.16_6. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting exasw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS ] Load firmware 2.2.16_6 onto exasw-iba01
[SUCCESS ] Verify that /conf/configvalid is set to 1 on exasw-iba01
[INFO ] Set SMPriority to 5 on exasw-iba01
[INFO ] Starting post-update validation on exasw-iba01
[SUCCESS ] Service opensmd is running on InfiniBand Switch exasw-iba01
[SUCCESS ] NTP daemon is running on exasw-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2023-03-09 Time:(HH:MM:SS) 11:26:30
[INFO ] /conf/configvalid is 1
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exasw-iba01
[SUCCESS ] Execute plugin check for Post Patch on exasw-iba01
[INFO ] Finished post-update validation on exasw-iba01
[SUCCESS ] Post-update validation on exasw-iba01
[SUCCESS ] SMPartition information is consistent on IB Switch exasw-iba01 post upgrade
[SUCCESS ] Update InfiniBand switch exasw-iba01 to 2.2.16_6
[INFO ] InfiniBand Switches ( exasw-iba01 exasw-ibb01 ) updated to 2.2.16_6
[SUCCESS ] Overall status
|||||
----- InfiniBand switch update process ended 2023-03-09 11:26:46 +0100 -----
2023-03-09 11:26:56 +0100 1 of 1 :SUCCESS: Upgrade InfiniBand switch(es) to 2.2.16-6.
2023-03-09 11:26:57 +0100 :SUCCESS: Completed run of command: ./patch_switch_23.1.0.0.0.230225.1/patchmgr -ibswitches /root/ibs_group -upgrade -rolling
2023-03-09 11:26:57 +0100 :INFO : upgrade performed on switch(es) in file /root/ibs_group: [exasw-iba01 exasw-ibb01]
2023-03-09 11:26:57 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/switch/patch_switch_23.1.0.0.0.230225.1:
2023-03-09 11:26:57 +0100 :INFO : - upgradeIBSwitch.log
2023-03-09 11:26:57 +0100 :INFO : - upgradeIBSwitch.trc
2023-03-09 11:26:57 +0100 :INFO : - patchmgr.stdout
2023-03-09 11:26:57 +0100 :INFO : - patchmgr.stderr
2023-03-09 11:26:57 +0100 :INFO : - patchmgr.log
2023-03-09 11:26:57 +0100 :INFO : - patchmgr.trc
2023-03-09 11:26:57 +0100 :INFO : Exit status:0
2023-03-09 11:26:57 +0100 :INFO : Exiting.
[DOM0 - root@exadom0m01 switch]$
As you can see, the patch was smooth and I running at version 2.2.16_6, no errors were reported, and were done one switch after another.
Patch Storage Cells
The process to patch the storage cell is really simple, we just need to download one file and call the patch. The patchmgr is already included and we just need to call it. But before we start is important to check and execute some steps.
First create one file with the hostname for all cells that you will patch:
[DOM0 - root@exadom0m01 ~]$ cat /root/all_cell exaceladm01 exaceladm02 exaceladm03 exaceladm04 exaceladm05 [DOM0 - root@exadom0m01 ~]$
After that we can check if we have some HW errors:
[DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/all_cell "ipmitool sunoem led get" |grep ": SERVICE" exaceladm01: SERVICE | OFF exaceladm02: SERVICE | OFF exaceladm03: SERVICE | OFF exaceladm04: SERVICE | OFF exaceladm05: SERVICE | OFF [DOM0 - root@exadom0m01 ~]$
And any other errors from the cell level. I recommend as well check quarantine sqls at the cell level (if you have, you can delete it because probably the related bug is solved in the new version. Use the command “drop quarantine all”):
[DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/all_cell cellcli -e "list alerthistory" [DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/all_cell cellcli -e "list quarantine" [DOM0 - root@exadom0m01 ~]$
After that we can reset the ILOM (old hardware can hide errors that come after just ILOM restart) and check again for HW errors:
[DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/all_cell "ipmitool bmc reset cold" exaceladm01: Sent cold reset command to MC exaceladm02: Sent cold reset command to MC exaceladm03: Sent cold reset command to MC exaceladm04: Sent cold reset command to MC exaceladm05: Sent cold reset command to MC [DOM0 - root@exadom0m01 ~]$ [DOM0 - root@exadom0m01 ~]$ dcli -l root -g /root/all_cell "ipmitool sunoem led get" |grep ": SERVICE" exaceladm01: SERVICE | OFF exaceladm02: SERVICE | OFF exaceladm03: SERVICE | OFF exaceladm04: SERVICE | OFF exaceladm05: SERVICE | OFF [DOM0 - root@exadom0m01 ~]$
Now we can unzip the patch over the base folder:
[DOM0 - root@exadom0m01 ~]$ cd /EXAVMIMAGES/patches/23.1.0/cell/ [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ ls -l total 1665024 -rw-r--r-- 1 root root 1704563354 Mar 9 08:46 p32829291_231000_Linux-x86-64.zip [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ unzip -q p32829291_231000_Linux-x86-64.zip [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ ls -l total 1665028 -rw-r--r-- 1 root root 1704563354 Mar 9 08:46 p32829291_231000_Linux-x86-64.zip drwxrwxr-x 5 root root 3896 Feb 26 00:38 patch_23.1.0.0.0.230225.1 [DOM0 - root@exadom0m01 cell]$
After that, we need to check if our cells have we can clean DSS keys from authorized_keys. With the new versions, you receive a warning during the precheck and fail during the patch time. So, we can clean each cell (for example, how to do in one cell, change the DSS for the public RSA key):
… … 2023-03-10 08:48:31 +0100 :Working: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ... 2023-03-10 08:48:37 +0100 :WARNING: For upgrades, patchmgr requires SSH equivalency without using DSA key to all cells. 2023-03-10 08:48:37 +0100 :WARNING: File /root/.ssh/authorized_keys on exaceladm01 has dsa keys that need to be removed prior updating. … … [DOM0 - root@exadom0m01 cell]$ cat ~root/.ssh/id_rsa.pub ssh-rsa AAAAB3NuMXiw9Tu+UQ== root@exadom0m01.simon.mynet.lu [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ ssh root@exaceladm01 Last login: Fri Mar 10 08:53:32 CET 2023 from exadom0m01.simon.mynet.lu on pts/0 Last login: Fri Mar 10 08:53:38 2023 from exadom0m01.simon.mynet.lu [root@exaceladm01 ~]# [root@exaceladm01 ~]# [root@exaceladm01 ~]# cat .ssh/authorized_keys |grep dss ssh-dss AAAAB3NzaOKFC/+bOv/uTgxeR4zrBTam0jYmweo= root@exadom0m01.simon.mynet.lu [root@exaceladm01 ~]# [root@exaceladm01 ~]# vi .ssh/authorized_keys [root@exaceladm01 ~]# [root@exaceladm01 ~]# cat .ssh/authorized_keys |grep dss [root@exaceladm01 ~]# [root@exaceladm01 ~]# cat .ssh/authorized_keys |grep exadom0m01 ssh-rsa AAAAB3NuMXiw9Tu+UQ== root@exadom0m01.simon.mynet.lu [root@exaceladm01 ~]# [root@exaceladm01 ~]# logout Connection to exaceladm01 closed. [DOM0 - root@exadom0m01 cell]$
Before doing the precheck one detail here. My actual version (for the cells as you can see below) is the last of 22.1.9.0.0.230302. And you can see that this was “packaged” at 230302. The new version 23.1 was created before this date. So, we can receive one error during the precheck because it is “newer” than the image that we are trying to install:
... exaceladm01: exaceladm01 160.10.192.32 2023-03-10 08:40:39 +0100: The target version needs to be higher than the current version. The target version date needs to be newer than the current version date. Newer dated releases contain more up-to-date content. The current version date, 230302, is newer than the target version date, 230225. ...
So, as you can see the 23.1 was “packaged” at 230225 and the precheck will complain. But this is easy to be solved, you just need to add the option “ignore_date_validations” when you call the patchmgr:
[DOM0 - root@exadom0m01 cell]$ ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -patch_check_prereq --ignore_date_validations 2023-03-10 08:54:49 +0100 :Working: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ... 2023-03-10 08:54:56 +0100 :SUCCESS: Check cells have ssh equivalence for root user. 2023-03-10 08:55:05 +0100 :Working: Check if upgrade is supported. 2023-03-10 08:55:09 +0100 :SUCCESS: Check if upgrade is supported. 2023-03-10 08:55:09 +0100 :Working: Initialize files. Up to 1 minute ... 2023-03-10 08:55:14 +0100 :Working: Setup work directory 2023-03-10 08:56:02 +0100 :SUCCESS: Setup work directory 2023-03-10 08:56:10 +0100 :SUCCESS: Initialize files. 2023-03-10 08:56:11 +0100 :Working: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ... 2023-03-10 08:56:31 +0100 :INFO : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes. 2023-03-10 08:56:40 +0100 :SUCCESS: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. 2023-03-10 08:56:40 +0100 :Working: Performing cell prechecks. Up to 20 minutes ... 2023-03-10 08:57:43 +0100 :SUCCESS: Performing cell prechecks. 2023-03-10 08:57:43 +0100 :Working: Check prerequisites on all cells. Up to 2 minutes ... 2023-03-10 08:57:59 +0100 :SUCCESS: Check prerequisites on all cells. 2023-03-10 08:57:59 +0100 :Working: Execute plugin check for Patch Check Prereq ... 2023-03-10 08:57:59 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0. 2023-03-10 08:57:59 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 08:57:59 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. 2023-03-10 08:57:59 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 08:57:59 +0100 :SUCCESS: No exposure to bug 17854520 with non-rolling patching 2023-03-10 08:57:59 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0. 2023-03-10 08:57:59 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 08:58:00 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216 2023-03-10 08:58:00 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0. 2023-03-10 08:58:00 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 08:58:00 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612 2023-03-10 08:58:00 +0100 :SUCCESS: No exposure to bug 22896791 with non-rolling patching 2023-03-10 08:58:00 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0. 2023-03-10 08:58:00 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 08:58:03 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315 2023-03-10 08:58:05 +0100 :SUCCESS: Execute plugin check for Patch Check Prereq. 2023-03-10 08:58:16 +0100 :Working: Check ASM deactivation outcome. Up to 1 minute ... 2023-03-10 08:58:28 +0100 :SUCCESS: Check ASM deactivation outcome. 2023-03-10 08:58:28 +0100 :Working: check if MS SOFTWAREUPDATE is scheduled. up to 5 minutes... 2023-03-10 08:58:29 +0100 :NO ACTION NEEDED: No cells found with SOFTWAREUPDATE scheduled by MS 2023-03-10 08:58:31 +0100 :SUCCESS: check if MS SOFTWAREUPDATE is scheduled 2023-03-10 08:58:31 +0100 :Working: Check resilvering status of griddisks. Up to 1 minute ... 2023-03-10 08:58:44 +0100 :SUCCESS: Check resilvering status of griddisks. 2023-03-10 08:58:44 +0100 :Working: check unsanctioned hidden parameter on the cell(s)... 2023-03-10 08:59:06 +0100 :NO ACTION NEEDED: No unsanctioned hidden parameter found on the cell(s) 2023-03-10 08:59:08 +0100 :SUCCESS: check unsanctioned hidden parameter on the cell(s) 2023-03-10 08:59:10 +0100 :SUCCESS: Completed run of command: ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -patch_check_prereq --ignore_date_validations 2023-03-10 08:59:10 +0100 :INFO : patch_prereq performed on cell(s) in file /root/all_cell: [exaceladm01 exaceladm02 exaceladm03 exaceladm04 exaceladm05] 2023-03-10 08:59:10 +0100 :INFO : Current image version on cell(s) is: 2023-03-10 08:59:10 +0100 :INFO : exaceladm01: 22.1.9.0.0.230302 2023-03-10 08:59:10 +0100 :INFO : exaceladm02: 22.1.9.0.0.230302 2023-03-10 08:59:10 +0100 :INFO : exaceladm03: 22.1.9.0.0.230302 2023-03-10 08:59:10 +0100 :INFO : exaceladm04: 22.1.9.0.0.230302 2023-03-10 08:59:10 +0100 :INFO : exaceladm05: 22.1.9.0.0.230302 2023-03-10 08:59:10 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1: 2023-03-10 08:59:10 +0100 :INFO : - <cell_name>.log 2023-03-10 08:59:11 +0100 :INFO : - patchmgr.stdout 2023-03-10 08:59:11 +0100 :INFO : - patchmgr.stderr 2023-03-10 08:59:11 +0100 :INFO : - patchmgr.log 2023-03-10 08:59:11 +0100 :INFO : - patchmgr.trc 2023-03-10 08:59:11 +0100 :INFO : Exit status:0 2023-03-10 08:59:11 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 cell]$
So, with success, we can upgrade the cells. But one point here. Bellow, I am patching in NO ROLLING mode (look that “rolling” parameter was not passed). This means that all my environment is down (all dbs, and domU’s). So, when you need to patch, please choose correctly the mode that you will do:
[DOM0 - root@exadom0m01 cell]$ screen -L -RR Patch-Cells [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -patch --ignore_date_validations ******************************************************************************** NOTE Cells will reboot during the patch or rollback process. NOTE For non-rolling patch or rollback, ensure all ASM instances using NOTE the cells are shut down for the duration of the patch or rollback. NOTE For rolling patch or rollback, ensure all ASM instances using NOTE the cells are up for the duration of the patch or rollback. WARNING Do not interrupt the patchmgr session. WARNING Do not alter state of ASM instances during patch or rollback. WARNING Do not resize the screen. It may disturb the screen layout. WARNING Do not reboot cells or alter cell services during patch or rollback. WARNING Do not open log files in editor in write mode or try to alter them. NOTE All time estimates are approximate. ******************************************************************************** 2023-03-10 08:59:53 +0100 :Working: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ... 2023-03-10 09:00:00 +0100 :SUCCESS: Check cells have ssh equivalence for root user. 2023-03-10 09:00:09 +0100 :Working: Check if upgrade is supported. 2023-03-10 09:00:13 +0100 :SUCCESS: Check if upgrade is supported. 2023-03-10 09:00:14 +0100 :Working: Initialize files. Up to 1 minute ... 2023-03-10 09:00:18 +0100 :Working: Setup work directory 2023-03-10 09:01:07 +0100 :SUCCESS: Setup work directory 2023-03-10 09:01:16 +0100 :SUCCESS: Initialize files. 2023-03-10 09:01:16 +0100 :Working: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ... 2023-03-10 09:01:36 +0100 :INFO : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes. 2023-03-10 09:01:45 +0100 :SUCCESS: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. 2023-03-10 09:01:45 +0100 :Working: Performing cell prechecks. Up to 20 minutes ... 2023-03-10 09:02:46 +0100 :SUCCESS: Performing cell prechecks. 2023-03-10 09:02:46 +0100 :Working: Check prerequisites on all cells. Up to 2 minutes ... 2023-03-10 09:03:01 +0100 :SUCCESS: Check prerequisites on all cells. 2023-03-10 09:03:01 +0100 :Working: Copy the patch to all cells. Up to 3 minutes ... 2023-03-10 09:04:23 +0100 :SUCCESS: Copy the patch to all cells. 2023-03-10 09:04:25 +0100 :Working: Execute plugin check for Patch Check Prereq ... 2023-03-10 09:04:25 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0. 2023-03-10 09:04:25 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 09:04:25 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. 2023-03-10 09:04:25 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 09:04:25 +0100 :SUCCESS: No exposure to bug 17854520 with non-rolling patching 2023-03-10 09:04:25 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0. 2023-03-10 09:04:25 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 09:04:26 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216 2023-03-10 09:04:26 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0. 2023-03-10 09:04:26 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 09:04:26 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612 2023-03-10 09:04:26 +0100 :SUCCESS: No exposure to bug 22896791 with non-rolling patching 2023-03-10 09:04:26 +0100 :INFO : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0. 2023-03-10 09:04:26 +0100 :INFO : Details in logfile /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1/patchmgr.stdout. 2023-03-10 09:04:29 +0100 :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315 2023-03-10 09:04:30 +0100 :SUCCESS: Execute plugin check for Patch Check Prereq. 2023-03-10 09:04:31 +0100 :Working: check if MS SOFTWAREUPDATE is scheduled. up to 5 minutes... 2023-03-10 09:04:33 +0100 :NO ACTION NEEDED: No cells found with SOFTWAREUPDATE scheduled by MS 2023-03-10 09:04:35 +0100 :SUCCESS: check if MS SOFTWAREUPDATE is scheduled 2023-03-10 09:04:35 +0100 :Working: check unsanctioned hidden parameter on the cell(s)... 2023-03-10 09:04:57 +0100 :NO ACTION NEEDED: No unsanctioned hidden parameter found on the cell(s) 2023-03-10 09:04:59 +0100 :SUCCESS: check unsanctioned hidden parameter on the cell(s) 2023-03-10 09:05:06 +0100 1 of 5 :Working: Initiate patch on cells. Cells will remain up. Up to 5 minutes ... 2023-03-10 09:05:22 +0100 1 of 5 :SUCCESS: Initiate patch on cells. 2023-03-10 09:05:22 +0100 2 of 5 :Working: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ... 2023-03-10 09:06:22 +0100 :INFO : Wait for patch pre-reboot procedures 2023-03-10 09:07:45 +0100 2 of 5 :SUCCESS: Waiting to finish pre-reboot patch actions. 2023-03-10 09:07:45 +0100 :Working: Execute plugin check for Patching ... 2023-03-10 09:07:47 +0100 :SUCCESS: Execute plugin check for Patching. 2023-03-10 09:07:47 +0100 3 of 5 :Working: Finalize patch on cells. Cells will reboot. Up to 5 minutes ... 2023-03-10 09:08:16 +0100 3 of 5 :SUCCESS: Finalize patch on cells. 2023-03-10 09:08:36 +0100 4 of 5 :Working: Wait for cells to reboot and come online. Up to 50 minutes ... 2023-03-10 09:09:36 +0100 :INFO : Wait for patch finalization and reboot 2023-03-10 09:32:14 +0100 4 of 5 :SUCCESS: Wait for cells to reboot and come online. 2023-03-10 09:32:14 +0100 5 of 5 :Working: Check the state of patch on cells. Up to 5 minutes ... 2023-03-10 09:33:37 +0100 5 of 5 :SUCCESS: Check the state of patch on cells. 2023-03-10 09:33:37 +0100 :Working: Execute plugin check for Pre Disk Activation ... 2023-03-10 09:33:38 +0100 :SUCCESS: Execute plugin check for Pre Disk Activation. 2023-03-10 09:33:38 +0100 :Working: Activate grid disks... 2023-03-10 09:33:45 +0100 :INFO : Wait for checking and activating grid disks 2023-03-10 09:34:05 +0100 :SUCCESS: Activate grid disks. 2023-03-10 09:34:12 +0100 :Working: Execute plugin check for Post Patch ... 2023-03-10 09:34:15 +0100 :SUCCESS: Execute plugin check for Post Patch. 2023-03-10 09:34:18 +0100 :Working: Cleanup 2023-03-10 09:35:11 +0100 :SUCCESS: Cleanup 2023-03-10 09:35:14 +0100 :SUCCESS: Completed run of command: ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -patch --ignore_date_validations 2023-03-10 09:35:14 +0100 :INFO : patch performed on cell(s) in file /root/all_cell: [exaceladm01 exaceladm02 exaceladm03 exaceladm04 exaceladm05] 2023-03-10 09:35:14 +0100 :INFO : Current image version on cell(s) is: 2023-03-10 09:35:14 +0100 :INFO : exaceladm01: 23.1.0.0.0.230225.1 2023-03-10 09:35:14 +0100 :INFO : exaceladm02: 23.1.0.0.0.230225.1 2023-03-10 09:35:14 +0100 :INFO : exaceladm03: 23.1.0.0.0.230225.1 2023-03-10 09:35:14 +0100 :INFO : exaceladm04: 23.1.0.0.0.230225.1 2023-03-10 09:35:14 +0100 :INFO : exaceladm05: 23.1.0.0.0.230225.1 2023-03-10 09:35:14 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1: 2023-03-10 09:35:14 +0100 :INFO : - <cell_name>.log 2023-03-10 09:35:14 +0100 :INFO : - patchmgr.stdout 2023-03-10 09:35:14 +0100 :INFO : - patchmgr.stderr 2023-03-10 09:35:14 +0100 :INFO : - patchmgr.log 2023-03-10 09:35:14 +0100 :INFO : - patchmgr.trc 2023-03-10 09:35:41 +0100 :INFO : Exit status:0 2023-03-10 09:35:41 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 cell]$
As you can see above, the patch was successful and everything was OK. In the beginning, you can see that I called inside of screen session.
After the success, you can cleanup (this will remove old tmp/unnecessary files):
[DOM0 - root@exadom0m01 cell]$ ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -cleanup 2023-03-10 09:38:58 +0100 :Working: Cleanup 2023-03-10 09:39:03 +0100 :SUCCESS: Cleanup 2023-03-10 09:39:06 +0100 :SUCCESS: Completed run of command: ./patch_23.1.0.0.0.230225.1/patchmgr -cells /root/all_cell -cleanup 2023-03-10 09:39:06 +0100 :INFO : Cleanup performed on cell(s) in file /root/all_cell: [exaceladm01 exaceladm02 exaceladm03 exaceladm04 exaceladm05] 2023-03-10 09:39:06 +0100 :INFO : Current image version on cell(s) is: 2023-03-10 09:39:06 +0100 :INFO : exaceladm01: 23.1.0.0.0.230225.1 2023-03-10 09:39:06 +0100 :INFO : exaceladm02: 23.1.0.0.0.230225.1 2023-03-10 09:39:06 +0100 :INFO : exaceladm03: 23.1.0.0.0.230225.1 2023-03-10 09:39:06 +0100 :INFO : exaceladm04: 23.1.0.0.0.230225.1 2023-03-10 09:39:06 +0100 :INFO : exaceladm05: 23.1.0.0.0.230225.1 2023-03-10 09:39:06 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/patches/23.1.0/cell/patch_23.1.0.0.0.230225.1: 2023-03-10 09:39:06 +0100 :INFO : - <cell_name>.log 2023-03-10 09:39:06 +0100 :INFO : - patchmgr.stdout 2023-03-10 09:39:06 +0100 :INFO : - patchmgr.stderr 2023-03-10 09:39:06 +0100 :INFO : - patchmgr.log 2023-03-10 09:39:06 +0100 :INFO : - patchmgr.trc 2023-03-10 09:39:06 +0100 :INFO : Exit status:0 2023-03-10 09:39:06 +0100 :INFO : Exiting. [DOM0 - root@exadom0m01 cell]$
And the new storage cell version is active:
[DOM0 - root@exadom0m01 cell]$ dcli -l root -g /root/all_cell "imageinfo" |grep "Active image version" exaceladm01: Active image version: 23.1.0.0.0.230225.1 exaceladm02: Active image version: 23.1.0.0.0.230225.1 exaceladm03: Active image version: 23.1.0.0.0.230225.1 exaceladm04: Active image version: 23.1.0.0.0.230225.1 exaceladm05: Active image version: 23.1.0.0.0.230225.1 [DOM0 - root@exadom0m01 cell]$ [DOM0 - root@exadom0m01 cell]$ dcli -l root -g /root/all_cell "cat /etc/redhat-release" exaceladm01: Red Hat Enterprise Linux release 8.7 (Ootpa) exaceladm02: Red Hat Enterprise Linux release 8.7 (Ootpa) exaceladm03: Red Hat Enterprise Linux release 8.7 (Ootpa) exaceladm04: Red Hat Enterprise Linux release 8.7 (Ootpa) exaceladm05: Red Hat Enterprise Linux release 8.7 (Ootpa) [DOM0 - root@exadom0m01 cell]$
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”