The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.
The fourth part covers the upgrade from 19.7 to 19.8. I separate so you can use this as a direct guide from 19.6 to 19.7 if you need to do just this update. Parts of this post are similar to the upgrade from 19.6 to 19.7 that I described in the previous post.
Patch Matrix
The matrix of what can be done can be found at this post from ODA blog, and you can check below:
Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.
Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.
Actual environment
The environment is ODA X5-2, running the version 19.7. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it is running bare metal (BM).
The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.
I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):
[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/ cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles' cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig' [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/ [root@odat1 ~]#
Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component. If we need we just need to update the components with odacli update-storage. This will (if some firmware is applied) reboot the ODA.
Before starting the patch I recommend stopping all databases running at ODA. This will speed up the process. So, left just GI and ASM running.
19.8
The upgrade from 19.7 to 19.8 is a simple process and follows a well-known procedure if you compare with the upgrade of 18.8 to 19.6. The 19.8 version enables some features, like OVM. But here I am doing the patch over BM and will continue to use this here. The patch to download is 31481816.
Update repository
The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):
[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli update-repository -f /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip { "jobId" : "0c4b1959-0f9c-4a41-9824-f7878a13719d", "status" : "Created", "message" : "/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip", "reports" : [ ], "createTimestamp" : "July 29, 2020 17:33:52 PM CEST", "resourceList" : [ ], "description" : "Repository Update", "updatedTime" : "July 29, 2020 17:33:52 PM CEST" } [root@odat1 19.8]#
After that we can follow the job with odacli describe-job:
[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli describe-job -i "0c4b1959-0f9c-4a41-9824-f7878a13719d" Job details ---------------------------------------------------------------- ID: 0c4b1959-0f9c-4a41-9824-f7878a13719d Description: Repository Update Status: Success Created: July 29, 2020 5:33:52 PM CEST Message: /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Check AvailableSpace July 29, 2020 5:33:53 PM CEST July 29, 2020 5:33:53 PM CEST Success Setting up ssh equivalance July 29, 2020 5:33:54 PM CEST July 29, 2020 5:33:54 PM CEST Success Copy BundleFile July 29, 2020 5:33:55 PM CEST July 29, 2020 5:38:07 PM CEST Success Validating CopiedFile July 29, 2020 5:38:07 PM CEST July 29, 2020 5:38:41 PM CEST Success Unzip bundle July 29, 2020 5:38:41 PM CEST July 29, 2020 5:56:35 PM CEST Success Unzip bundle July 29, 2020 5:56:35 PM CEST July 29, 2020 5:58:36 PM CEST Success Delete PatchBundles July 29, 2020 5:58:43 PM CEST July 29, 2020 5:58:58 PM CEST Success Removing ssh keys July 29, 2020 5:58:59 PM CEST July 29, 2020 5:59:00 PM CEST Success [root@odat1 19.8]#
DCSAgent
The next step is to update the internal DCS agent for ODA 19.8. Is a simple step executed using the odacli update-dcsagent. The parameter is the version 19.8.0.0:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.8.0.0.0 { "jobId" : "a3ad9edf-f58d-40b0-9ea0-75e9347907c3", "status" : "Created", "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands", "reports" : [ ], "createTimestamp" : "August 06, 2020 17:30:46 PM CEST", "resourceList" : [ ], "description" : "DcsAgent patching", "updatedTime" : "August 06, 2020 17:30:46 PM CEST" } [root@odat1 ~]#
And as usual, we need to wait for the job to finish before continuing. We use the odacli describe-job to check this:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3" Job details ---------------------------------------------------------------- ID: a3ad9edf-f58d-40b0-9ea0-75e9347907c3 Description: DcsAgent patching Status: Success Created: August 6, 2020 5:30:46 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- dcs-agent upgrade to version 19.8.0.0.0 August 6, 2020 5:30:47 PM CEST August 6, 2020 5:32:59 PM CEST Success dcs-agent upgrade to version 19.8.0.0.0 August 6, 2020 5:32:59 PM CEST August 6, 2020 5:34:54 PM CEST Success Update System version August 6, 2020 5:35:00 PM CEST August 6, 2020 5:35:00 PM CEST Success Update System version August 6, 2020 5:35:01 PM CEST August 6, 2020 5:35:01 PM CEST Success [root@odat1 ~]#
You can receive some errors if you try to describe the jobID. This can occur because the process restarts the DCS. If you got this, just wait some minutes and try again (some common errors are: DCS-10001:Internal error encountered: Fail to get credential or DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3" DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070. [root@odat1 ~]#
DCSAdmin
The next step will be to update the dcsadmin using the command odacli update-dcsadmin:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.7.0.0.0 { "jobId" : "edf88f79-529c-438e-8a4a-a412749c8a70", "status" : "Created", "message" : null, "reports" : [ ], "createTimestamp" : "July 26, 2020 11:56:45 AM CEST", "resourceList" : [ ], "description" : "DcsAdmin patching", "updatedTime" : "July 26, 2020 11:56:45 AM CEST" } [root@odat1 ~]#
And as usual, we can check the end of the job before continue:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "e00b6dd8-d315-40de-b612-641d4853fe05" Job details ---------------------------------------------------------------- ID: e00b6dd8-d315-40de-b612-641d4853fe05 Description: DcsAdmin patching Status: Success Created: August 6, 2020 5:36:09 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation August 6, 2020 5:36:12 PM CEST August 6, 2020 5:36:12 PM CEST Success Patch location validation August 6, 2020 5:36:12 PM CEST August 6, 2020 5:36:12 PM CEST Success dcsadmin upgrade August 6, 2020 5:36:13 PM CEST August 6, 2020 5:36:21 PM CEST Success dcsadmin upgrade August 6, 2020 5:36:21 PM CEST August 6, 2020 5:36:29 PM CEST Success Update System version August 6, 2020 5:36:30 PM CEST August 6, 2020 5:36:30 PM CEST Success Update System version August 6, 2020 5:36:30 PM CEST August 6, 2020 5:36:30 PM CEST Success [root@odat1 ~]#
DCSComponents
The next is to update the dcscomponents with command odacli update-dcscomponents. As occurred with other versions, the commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.8.0.0.0 { "jobId" : "0f27ebd9-591f-4242-9e89-b998d93ecede", "status" : "Success", "message" : null, "reports" : null, "createTimestamp" : "August 06, 2020 17:37:01 PM CEST", "description" : "Job completed and is not part of Agent job list", "updatedTime" : "August 06, 2020 17:37:01 PM CEST" } [root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "0f27ebd9-591f-4242-9e89-b998d93ecede" DCS-10000:Resource Job with ID 0f27ebd9-591f-4242-9e89-b998d93ecede is not found. [root@odat1 ~]# [root@odat1 ~]# /opt/oracle/dcs/bin/odacli list-jobs ID Description Created Status ---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ---------- ... 0c4b1959-0f9c-4a41-9824-f7878a13719d Repository Update July 29, 2020 5:33:52 PM CEST Success a3ad9edf-f58d-40b0-9ea0-75e9347907c3 DcsAgent patching August 6, 2020 5:30:46 PM CEST Success e00b6dd8-d315-40de-b612-641d4853fe05 DcsAdmin patching August 6, 2020 5:36:09 PM CEST Success cae0726b-3298-4c02-ba8e-b4be76eab620 SSH keys update August 6, 2020 5:37:24 PM CEST Success a702447e-71ed-42af-904c-1e3225f193df SSH key delete August 6, 2020 5:37:29 PM CEST Success 348599c5-2718-40fa-8bbf-f3ecfee28123 SSH keys update August 6, 2020 5:37:59 PM CEST Success cff63e2a-be64-4e2d-9500-f274d6c2a640 SSH key delete August 6, 2020 5:39:33 PM CEST Success [root@odat1 ~]#
Create-prepatchreport
The next step is to verify if you can update it to 19.8. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.7.0.0.0 Job details ---------------------------------------------------------------- ID: d7d1c155-1312-41bf-afe6-c85570073d9b Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: Created Created: July 26, 2020 1:59:04 PM CEST Message: Use 'odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b' to check details of results Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- [root@odat1 ~]#
And after some time we got the full result (30 minutes more or less):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e150f6d2-19c2-40c9-90da-3d427234d1b6 Patch pre-check report ------------------------------------------------------------------------ Job ID: e150f6d2-19c2-40c9-90da-3d427234d1b6 Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: FAILED Created: August 6, 2020 6:28:15 PM CEST Result: One or more pre-checks failed for [ORACHK] Node Name --------------- odat1 Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Verify OS patch Success Verified OS patch Validate command execution Success Validated command execution __ILOM__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Checking Ilom patch Version Success Successfully verified the versions Patch location validation Success Successfully validated location Validate command execution Success Validated command execution __GI__ Validate supported GI versions Success Validated minimum supported versions. Validate available space Success Validated free space under /u01 Is clusterware running Success Clusterware is running Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is system provisioned Success Verified system is provisioned Validate ASM in online Success ASM is online Validate minimum agent version Success GI patching enabled in current DCSAGENT version Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0. Validate clones location exist Success Validated clones location Is patch location available Success Patch location is available. Patch location validation Success Successfully validated location Patch verification Success Patches 31281355 not applied on GI home /u01/app/19.0.0.0/grid on node odat1 Validate Opatch update Success Successfully updated the opatch in GiHome /u01/app/19.0.0.0/grid on node odat1 Patch conflict check Success No patch conflicts found on GiHome /u01/app/19.0.0.0/grid on node odat1 Validate command execution Success Validated command execution __ORACHK__ Running orachk Failed Orachk validation failed: . Validate command execution Success Validated command execution Software home Failed Software home check failed Node Name --------------- odat2 Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Verify OS patch Success Verified OS patch Validate command execution Success Validated command execution __ILOM__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Checking Ilom patch Version Success Successfully verified the versions Patch location validation Success Successfully validated location Validate command execution Success Validated command execution __GI__ Validate supported GI versions Success Validated minimum supported versions. Validate available space Success Validated free space under /u01 Is clusterware running Success Clusterware is running Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is system provisioned Success Verified system is provisioned Validate ASM in online Success ASM is online Validate minimum agent version Success GI patching enabled in current DCSAGENT version Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0. Validate clones location exist Success Validated clones location Is patch location available Success Patch location is available. Patch location validation Success Successfully validated location Patch verification Success Patches 31281355 not applied on GI home /u01/app/19.0.0.0/grid on node odat2 Validate Opatch update Success Successfully updated the opatch in GiHome /u01/app/19.0.0.0/grid on node odat2 Patch conflict check Success No patch conflicts found on GiHome /u01/app/19.0.0.0/grid on node odat2 Validate command execution Success Validated command execution __ORACHK__ Running orachk Failed Orachk validation failed: . Validate command execution Success Validated command execution Software home Failed Software home check failed [root@odat1 ~]#
As you can see FAILED. Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success in the report.
The ORACHK is more tricky to check the source. To check we need go to folder /opt/oracle/dcs/oracle.ahf/orachk/SERVER/<jobID#>/orachk_<hostname><random#>/ and check the HTML file:
[root@odat1 ~]# ls -l /opt/oracle/dcs/oracle.ahf/orachk/SERVER/d7d1c155-1312-41bf-afe6-c85570073d9b/orachk_odat1_072620_14087/ total 1024 drwxr-xr-x 2 root root 4096 Jul 26 14:12 log -rw------- 1 root root 993400 Jul 26 14:12 orachk_odat1_072620_14087.html drwxr-xr-x 4 root root 24576 Jul 26 14:12 outfiles drwxr-xr-x 2 root root 4096 Jul 26 14:12 reports drwxr-xr-x 2 root root 12288 Jul 26 14:12 scripts drwxr-xr-x 2 root root 4096 Jul 26 14:12 upload [root@odat1 ~]#
There we can see internally the ERROR. In this case that we got the “Orachk validation failed” and “Software home check failed” the error is related with a check made but CVU that try to check some files (that are listed in XML file) and was not found:
Status on odat1: FAIL => Software home check failed Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libjavafx_font_t2k.so" could not be verified on node "odat1". OS error: "No such file or directory" Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libkcms.so" could not be verified on node "odat1". OS error: "No such file or directory"
This error occurs because the CVU uses the context of file /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml to verify the files that need to be checked. But this file has more contents that need:
[root@odat1 ~]# cat /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml |grep libkcms <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/> <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/> <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/> [root@odat1 ~]#
So, to fix, need to copy the file /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml over /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml. This is described at Bug 31208297 since the new version is aligned with 19c (while the other is aligned with other versions):
[root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml /tmp [root@odat1 ~]# [root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml -rw-r--r-- 1 grid oinstall 3719189 Apr 7 2019 /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml [root@odat1 ~]# [root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml -rw-r--r-- 1 grid oinstall 3723425 Apr 16 09:29 /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml cp: overwrite ‘/u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml’? y [root@odat1 ~]#
Another option is simply to ignore the error and continue with the process. If you have doubts or other errors, please check the release notes from the version to check if they are listed there. Another option is open SR.
After fixing the orachk we can run the patchreport again:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.8.0.0.0 Job details ---------------------------------------------------------------- ID: e2bfcfbf-3925-40f7-b666-2e55c0d77f68 Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: Created Created: August 6, 2020 7:01:30 PM CEST Message: Use 'odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68' to check details of results Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- [root@odat1 ~]# [root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68 Patch pre-check report ------------------------------------------------------------------------ Job ID: e2bfcfbf-3925-40f7-b666-2e55c0d77f68 Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: SUCCESS Created: August 6, 2020 7:01:30 PM CEST Result: All pre-checks succeeded Node Name --------------- odat1 Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Verify OS patch Success Verified OS patch Validate command execution Success Validated command execution __ILOM__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Checking Ilom patch Version Success Patch already applied Patch location validation Success Successfully validated location Validate command execution Success Validated command execution __GI__ Validate supported GI versions Success Validated minimum supported versions. Validate available space Success Validated free space under /u01 Is clusterware running Success Clusterware is running Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is system provisioned Success Verified system is provisioned Validate ASM in online Success ASM is online Validate minimum agent version Success GI patching enabled in current DCSAGENT version Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0. Validate clones location exist Success Validated clones location Is patch location available Success Patch location is available. Patch location validation Success Successfully validated location Patch verification Success Patches 31281355 not applied on GI home /u01/app/19.0.0.0/grid on node odat1 Validate Opatch update Success Successfully updated the opatch in GiHome /u01/app/19.0.0.0/grid on node odat1 Patch conflict check Success No patch conflicts found on GiHome /u01/app/19.0.0.0/grid on node odat1 Validate command execution Success Validated command execution __ORACHK__ Running orachk Success Succesfully ran Orachk Validate command execution Success Validated command execution Node Name --------------- odat2 Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Verify OS patch Success Verified OS patch Validate command execution Success Validated command execution __ILOM__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Checking Ilom patch Version Success Patch already applied Patch location validation Success Successfully validated location Validate command execution Success Validated command execution __GI__ Validate supported GI versions Success Validated minimum supported versions. Validate available space Success Validated free space under /u01 Is clusterware running Success Clusterware is running Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is system provisioned Success Verified system is provisioned Validate ASM in online Success ASM is online Validate minimum agent version Success GI patching enabled in current DCSAGENT version Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0. Validate clones location exist Success Validated clones location Is patch location available Success Patch location is available. Patch location validation Success Successfully validated location Patch verification Success Patches 31281355 not applied on GI home /u01/app/19.0.0.0/grid on node odat2 Validate Opatch update Success Successfully updated the opatch in GiHome /u01/app/19.0.0.0/grid on node odat2 Patch conflict check Success No patch conflicts found on GiHome /u01/app/19.0.0.0/grid on node odat2 Validate command execution Success Validated command execution __ORACHK__ Running orachk Success Succesfully ran Orachk Validate command execution Success Validated command execution [root@odat1 ~]#
Update-Server
After the create patchreport we can patch the server calling odacli update-server:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-server -v 19.8.0.0.0 { "jobId" : "58472ea7-f329-45d5-a8c0-e13cf9074f5f", "status" : "Created", "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.", "reports" : [ ], "createTimestamp" : "August 06, 2020 19:22:24 PM CEST", "resourceList" : [ ], "description" : "Server Patching", "updatedTime" : "August 06, 2020 19:22:24 PM CEST" } [root@odat1 ~]#
This will create one jobIB that we can follow:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "58472ea7-f329-45d5-a8c0-e13cf9074f5f" Job details ---------------------------------------------------------------- ID: 58472ea7-f329-45d5-a8c0-e13cf9074f5f Description: Server Patching Status: Success Created: August 6, 2020 7:22:24 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation August 6, 2020 7:23:04 PM CEST August 6, 2020 7:23:05 PM CEST Success Patch location validation August 6, 2020 7:23:04 PM CEST August 6, 2020 7:23:05 PM CEST Success dcs-controller upgrade August 6, 2020 7:23:08 PM CEST August 6, 2020 7:23:21 PM CEST Success dcs-controller upgrade August 6, 2020 7:23:22 PM CEST August 6, 2020 7:23:36 PM CEST Success Patch location validation August 6, 2020 7:23:38 PM CEST August 6, 2020 7:23:38 PM CEST Success Patch location validation August 6, 2020 7:23:38 PM CEST August 6, 2020 7:23:38 PM CEST Success dcs-cli upgrade August 6, 2020 7:23:39 PM CEST August 6, 2020 7:23:40 PM CEST Success dcs-cli upgrade August 6, 2020 7:23:40 PM CEST August 6, 2020 7:23:41 PM CEST Success Creating repositories using yum August 6, 2020 7:23:46 PM CEST August 6, 2020 7:23:51 PM CEST Success Updating YumPluginVersionLock rpm August 6, 2020 7:23:51 PM CEST August 6, 2020 7:23:51 PM CEST Success Applying OS Patches August 6, 2020 7:23:51 PM CEST August 6, 2020 7:30:47 PM CEST Success Creating repositories using yum August 6, 2020 7:30:48 PM CEST August 6, 2020 7:30:50 PM CEST Success Applying HMP Patches August 6, 2020 7:30:50 PM CEST August 6, 2020 7:31:17 PM CEST Success Patch location validation August 6, 2020 7:31:19 PM CEST August 6, 2020 7:31:19 PM CEST Success Patch location validation August 6, 2020 7:31:19 PM CEST August 6, 2020 7:31:19 PM CEST Success oda-hw-mgmt upgrade August 6, 2020 7:31:20 PM CEST August 6, 2020 7:31:54 PM CEST Success oda-hw-mgmt upgrade August 6, 2020 7:31:54 PM CEST August 6, 2020 7:32:30 PM CEST Success OSS Patching August 6, 2020 7:32:31 PM CEST August 6, 2020 7:32:32 PM CEST Success Applying Firmware Disk Patches August 6, 2020 7:33:30 PM CEST August 6, 2020 7:34:19 PM CEST Success Applying Firmware Expander Patches August 6, 2020 7:34:33 PM CEST August 6, 2020 7:34:50 PM CEST Success Applying Firmware Controller Patches August 6, 2020 7:35:02 PM CEST August 6, 2020 7:35:17 PM CEST Success Checking Ilom patch Version August 6, 2020 7:35:20 PM CEST August 6, 2020 7:35:22 PM CEST Success Checking Ilom patch Version August 6, 2020 7:35:22 PM CEST August 6, 2020 7:35:25 PM CEST Success Patch location validation August 6, 2020 7:35:25 PM CEST August 6, 2020 7:35:26 PM CEST Success Patch location validation August 6, 2020 7:35:25 PM CEST August 6, 2020 7:35:26 PM CEST Success Save password in Wallet August 6, 2020 7:35:28 PM CEST August 6, 2020 7:35:28 PM CEST Success Apply Ilom patch August 6, 2020 7:35:28 PM CEST August 6, 2020 7:35:29 PM CEST Success Apply Ilom patch August 6, 2020 7:35:30 PM CEST August 6, 2020 7:35:31 PM CEST Success Copying Flash Bios to Temp location August 6, 2020 7:35:31 PM CEST August 6, 2020 7:35:31 PM CEST Success Copying Flash Bios to Temp location August 6, 2020 7:35:31 PM CEST August 6, 2020 7:35:31 PM CEST Success Starting the clusterware August 6, 2020 7:37:41 PM CEST August 6, 2020 7:39:40 PM CEST Success clusterware patch verification August 6, 2020 7:39:46 PM CEST August 6, 2020 7:39:55 PM CEST Success clusterware patch verification August 6, 2020 7:39:46 PM CEST August 6, 2020 7:39:54 PM CEST Success Patch location validation August 6, 2020 7:39:55 PM CEST August 6, 2020 7:40:03 PM CEST Success Patch location validation August 6, 2020 7:39:55 PM CEST August 6, 2020 7:40:03 PM CEST Success Opatch update August 6, 2020 7:40:53 PM CEST August 6, 2020 7:40:58 PM CEST Success Opatch update August 6, 2020 7:40:53 PM CEST August 6, 2020 7:40:56 PM CEST Success Patch conflict check August 6, 2020 7:40:58 PM CEST August 6, 2020 7:42:09 PM CEST Success Patch conflict check August 6, 2020 7:42:09 PM CEST August 6, 2020 7:43:22 PM CEST Success clusterware upgrade August 6, 2020 7:43:54 PM CEST August 6, 2020 8:25:43 PM CEST Success clusterware upgrade August 6, 2020 8:25:43 PM CEST August 6, 2020 9:08:22 PM CEST Success Updating GiHome version August 6, 2020 9:08:22 PM CEST August 6, 2020 9:08:46 PM CEST Success Updating GiHome version August 6, 2020 9:08:22 PM CEST August 6, 2020 9:08:46 PM CEST Success Update System version August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:12 PM CEST Success Update System version August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:12 PM CEST Success preRebootNode Actions August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:58 PM CEST Success preRebootNode Actions August 6, 2020 9:09:58 PM CEST August 6, 2020 9:10:44 PM CEST Success Reboot Ilom August 6, 2020 9:10:44 PM CEST August 6, 2020 9:10:44 PM CEST Success Reboot Ilom August 6, 2020 9:10:44 PM CEST August 6, 2020 9:10:45 PM CEST Success [root@odat1 ~]#
In ODAX5-2 this process took almost two hours. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like and steps executed. Some example:
... 2020-08-06 19:43:24,793 INFO [Disabling TFA : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.z.DCSZQueue: Node not yet received from /nodes/node_1/cmd-out-q with prefix:850_2186_ 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : .---------------------------------------------------------------------------------------------. 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : +-------+---------------+-------+------+------------+----------------------+------------------+ 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | odat1 | RUNNING | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | odat2 | RUNNING | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : '-------+---------------+-------+------+------------+----------------------+------------------' 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.DCSProcessBuilder: Return code: 0 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.CommandExecutor: Return code: 0 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: TfaUtils: parseOutputToCheckIsTFARunning 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output: .---------------------------------------------------------------------------------------------. | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | +-------+---------------+-------+------+------------+----------------------+------------------+ | odat1 | RUNNING | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | | odat2 | RUNNING | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | '-------+---------------+-------+------+------------+----------------------+------------------' 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output by line: .---------------------------------------------------------------------------------------------. ...
After you finish this jobIB, and around 10 minutes later, ODA nodes will reboot to boot the new version.
Update-storage
The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not update the firmware of the disks as an example.
You can see from odacli describe-component that some part are not up to date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component System Version --------------- 19.8.0.0.0 System node Name --------------- odat1 Local System Version --------------- 19.8.0.0.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 19.8.0.0.0 up-to-date GI 19.8.0.0.200714 up-to-date DB { [ OraDB18000_home1 ] 18.3.0.0.180717 18.11.0.0.200714 [ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200714 [ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200714 } DCSAGENT 19.8.0.0.0 up-to-date ILOM 4.0.4.52.r132805 up-to-date BIOS 30300200 up-to-date OS 7.8 up-to-date FIRMWARECONTROLLER { [ c0 ] 4.650.00-7176 up-to-date [ c1,c2 ] 16.00.08.00 up-to-date } FIRMWAREEXPANDER 001E up-to-date FIRMWAREDISK { [ c0d0,c0d1 ] A7E0 up-to-date [ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13, c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4, c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11, c2d12,c2d13,c2d14,c2d15 ] [ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date c1d22,c1d23,c2d16,c2d17,c2d18,c2d19, c2d20,c2d21,c2d22,c2d23 ] } HMP 2.4.5.0.1 up-to-date System node Name --------------- odat2 Local System Version --------------- 19.8.0.0.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 19.8.0.0.0 up-to-date GI 19.8.0.0.200714 up-to-date DB { [ OraDB18000_home1 ] 18.3.0.0.180717 18.11.0.0.200714 [ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200714 [ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200714 } DCSAGENT 19.8.0.0.0 up-to-date ILOM 4.0.4.52.r132805 up-to-date BIOS 30300200 up-to-date OS 7.8 up-to-date FIRMWARECONTROLLER { [ c0 ] 4.650.00-7176 up-to-date [ c1,c2 ] 16.00.08.00 up-to-date } FIRMWAREEXPANDER 001E up-to-date FIRMWAREDISK { [ c0d0,c0d1 ] A7E0 up-to-date [ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13, c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4, c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11, c2d12,c2d13,c2d14,c2d15 ] [ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date c1d22,c1d23,c2d16,c2d17,c2d18,c2d19, c2d20,c2d21,c2d22,c2d23 ] } HMP 2.4.5.0.1 up-to-date [root@odat1 ~]#
Even if everything is up-to-date we can call odacli update-storage to patch everything:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.8.0.0.0 { "jobId" : "cf7e78b9-c476-4f65-964a-0126e9a4d1f3", "status" : "Created", "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart", "reports" : [ ], "createTimestamp" : "August 06, 2020 21:27:49 PM CEST", "resourceList" : [ ], "description" : "Storage Firmware Patching", "updatedTime" : "August 06, 2020 21:27:49 PM CEST" } [root@odat1 ~]#
And we can follow the jobID:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "cf7e78b9-c476-4f65-964a-0126e9a4d1f3" Job details ---------------------------------------------------------------- ID: cf7e78b9-c476-4f65-964a-0126e9a4d1f3 Description: Storage Firmware Patching Status: Success Created: August 6, 2020 9:27:49 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Applying Firmware Disk Patches August 6, 2020 9:32:08 PM CEST August 6, 2020 9:34:38 PM CEST Success Applying Firmware Controller Patches August 6, 2020 9:34:52 PM CEST August 6, 2020 9:35:07 PM CEST Success preRebootNode Actions August 6, 2020 9:35:08 PM CEST August 6, 2020 9:35:08 PM CEST Success preRebootNode Actions August 6, 2020 9:35:08 PM CEST August 6, 2020 9:35:08 PM CEST Success Reboot Ilom August 6, 2020 9:35:09 PM CEST August 6, 2020 9:35:09 PM CEST Success Reboot Ilom August 6, 2020 9:35:09 PM CEST August 6, 2020 9:35:09 PM CEST Success [root@odat1 ~]#
After it finishes, the ODA nodes will reboot again. To reboot be effective maybe you need to wait 10 minutes or more.
And after the reboot, you can see that everything is up-to-date using odacli update-storage.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: Upgrade AHF and TFA at ODA - Fernando Simon