The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.
The fourth part covers the upgrade from 19.7 to 19.8. I separate so you can use this as a direct guide from 19.6 to 19.7 if you need to do just this update. Parts of this post are similar to the upgrade from 19.6 to 19.7 that I described in the previous post.
Patch Matrix
The matrix of what can be done can be found at this post from ODA blog, and you can check below:
Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.
Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.
Actual environment
The environment is ODA X5-2, running the version 19.7. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it is running bare metal (BM).
The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.
I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):
[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1 [root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/ cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles' cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig' [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/ [root@odat1 ~]#
Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component. If we need we just need to update the components with odacli update-storage. This will (if some firmware is applied) reboot the ODA.
Before starting the patch I recommend stopping all databases running at ODA. This will speed up the process. So, left just GI and ASM running.
19.8
The upgrade from 19.7 to 19.8 is a simple process and follows a well-known procedure if you compare with the upgrade of 18.8 to 19.6. The 19.8 version enables some features, like OVM. But here I am doing the patch over BM and will continue to use this here. The patch to download is 31481816.
Update repository
The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):
[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli update-repository -f /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip
{
"jobId" : "0c4b1959-0f9c-4a41-9824-f7878a13719d",
"status" : "Created",
"message" : "/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip",
"reports" : [ ],
"createTimestamp" : "July 29, 2020 17:33:52 PM CEST",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "July 29, 2020 17:33:52 PM CEST"
}
[root@odat1 19.8]#
After that we can follow the job with odacli describe-job:
[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli describe-job -i "0c4b1959-0f9c-4a41-9824-f7878a13719d"
Job details
----------------------------------------------------------------
ID: 0c4b1959-0f9c-4a41-9824-f7878a13719d
Description: Repository Update
Status: Success
Created: July 29, 2020 5:33:52 PM CEST
Message: /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace July 29, 2020 5:33:53 PM CEST July 29, 2020 5:33:53 PM CEST Success
Setting up ssh equivalance July 29, 2020 5:33:54 PM CEST July 29, 2020 5:33:54 PM CEST Success
Copy BundleFile July 29, 2020 5:33:55 PM CEST July 29, 2020 5:38:07 PM CEST Success
Validating CopiedFile July 29, 2020 5:38:07 PM CEST July 29, 2020 5:38:41 PM CEST Success
Unzip bundle July 29, 2020 5:38:41 PM CEST July 29, 2020 5:56:35 PM CEST Success
Unzip bundle July 29, 2020 5:56:35 PM CEST July 29, 2020 5:58:36 PM CEST Success
Delete PatchBundles July 29, 2020 5:58:43 PM CEST July 29, 2020 5:58:58 PM CEST Success
Removing ssh keys July 29, 2020 5:58:59 PM CEST July 29, 2020 5:59:00 PM CEST Success
[root@odat1 19.8]#
DCSAgent
The next step is to update the internal DCS agent for ODA 19.8. Is a simple step executed using the odacli update-dcsagent. The parameter is the version 19.8.0.0:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.8.0.0.0
{
"jobId" : "a3ad9edf-f58d-40b0-9ea0-75e9347907c3",
"status" : "Created",
"message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
"reports" : [ ],
"createTimestamp" : "August 06, 2020 17:30:46 PM CEST",
"resourceList" : [ ],
"description" : "DcsAgent patching",
"updatedTime" : "August 06, 2020 17:30:46 PM CEST"
}
[root@odat1 ~]#
And as usual, we need to wait for the job to finish before continuing. We use the odacli describe-job to check this:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3"
Job details
----------------------------------------------------------------
ID: a3ad9edf-f58d-40b0-9ea0-75e9347907c3
Description: DcsAgent patching
Status: Success
Created: August 6, 2020 5:30:46 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade to version 19.8.0.0.0 August 6, 2020 5:30:47 PM CEST August 6, 2020 5:32:59 PM CEST Success
dcs-agent upgrade to version 19.8.0.0.0 August 6, 2020 5:32:59 PM CEST August 6, 2020 5:34:54 PM CEST Success
Update System version August 6, 2020 5:35:00 PM CEST August 6, 2020 5:35:00 PM CEST Success
Update System version August 6, 2020 5:35:01 PM CEST August 6, 2020 5:35:01 PM CEST Success
[root@odat1 ~]#
You can receive some errors if you try to describe the jobID. This can occur because the process restarts the DCS. If you got this, just wait some minutes and try again (some common errors are: DCS-10001:Internal error encountered: Fail to get credential or DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3" DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070. [root@odat1 ~]#
DCSAdmin
The next step will be to update the dcsadmin using the command odacli update-dcsadmin:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.7.0.0.0
{
"jobId" : "edf88f79-529c-438e-8a4a-a412749c8a70",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "July 26, 2020 11:56:45 AM CEST",
"resourceList" : [ ],
"description" : "DcsAdmin patching",
"updatedTime" : "July 26, 2020 11:56:45 AM CEST"
}
[root@odat1 ~]#
And as usual, we can check the end of the job before continue:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "e00b6dd8-d315-40de-b612-641d4853fe05"
Job details
----------------------------------------------------------------
ID: e00b6dd8-d315-40de-b612-641d4853fe05
Description: DcsAdmin patching
Status: Success
Created: August 6, 2020 5:36:09 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation August 6, 2020 5:36:12 PM CEST August 6, 2020 5:36:12 PM CEST Success
Patch location validation August 6, 2020 5:36:12 PM CEST August 6, 2020 5:36:12 PM CEST Success
dcsadmin upgrade August 6, 2020 5:36:13 PM CEST August 6, 2020 5:36:21 PM CEST Success
dcsadmin upgrade August 6, 2020 5:36:21 PM CEST August 6, 2020 5:36:29 PM CEST Success
Update System version August 6, 2020 5:36:30 PM CEST August 6, 2020 5:36:30 PM CEST Success
Update System version August 6, 2020 5:36:30 PM CEST August 6, 2020 5:36:30 PM CEST Success
[root@odat1 ~]#
DCSComponents
The next is to update the dcscomponents with command odacli update-dcscomponents. As occurred with other versions, the commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.8.0.0.0
{
"jobId" : "0f27ebd9-591f-4242-9e89-b998d93ecede",
"status" : "Success",
"message" : null,
"reports" : null,
"createTimestamp" : "August 06, 2020 17:37:01 PM CEST",
"description" : "Job completed and is not part of Agent job list",
"updatedTime" : "August 06, 2020 17:37:01 PM CEST"
}
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "0f27ebd9-591f-4242-9e89-b998d93ecede"
DCS-10000:Resource Job with ID 0f27ebd9-591f-4242-9e89-b998d93ecede is not found.
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli list-jobs
ID Description Created Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
...
0c4b1959-0f9c-4a41-9824-f7878a13719d Repository Update July 29, 2020 5:33:52 PM CEST Success
a3ad9edf-f58d-40b0-9ea0-75e9347907c3 DcsAgent patching August 6, 2020 5:30:46 PM CEST Success
e00b6dd8-d315-40de-b612-641d4853fe05 DcsAdmin patching August 6, 2020 5:36:09 PM CEST Success
cae0726b-3298-4c02-ba8e-b4be76eab620 SSH keys update August 6, 2020 5:37:24 PM CEST Success
a702447e-71ed-42af-904c-1e3225f193df SSH key delete August 6, 2020 5:37:29 PM CEST Success
348599c5-2718-40fa-8bbf-f3ecfee28123 SSH keys update August 6, 2020 5:37:59 PM CEST Success
cff63e2a-be64-4e2d-9500-f274d6c2a640 SSH key delete August 6, 2020 5:39:33 PM CEST Success
[root@odat1 ~]#
Create-prepatchreport
The next step is to verify if you can update it to 19.8. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.7.0.0.0
Job details
----------------------------------------------------------------
ID: d7d1c155-1312-41bf-afe6-c85570073d9b
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: Created
Created: July 26, 2020 1:59:04 PM CEST
Message: Use 'odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@odat1 ~]#
And after some time we got the full result (30 minutes more or less):
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e150f6d2-19c2-40c9-90da-3d427234d1b6
Patch pre-check report
------------------------------------------------------------------------
Job ID: e150f6d2-19c2-40c9-90da-3d427234d1b6
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: FAILED
Created: August 6, 2020 6:28:15 PM CEST
Result: One or more pre-checks failed for [ORACHK]
Node Name
---------------
odat1
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 31281355 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat1
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat1
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat1
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Failed Orachk validation failed: .
Validate command execution Success Validated command execution
Software home Failed Software home check failed
Node Name
---------------
odat2
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 31281355 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat2
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat2
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat2
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Failed Orachk validation failed: .
Validate command execution Success Validated command execution
Software home Failed Software home check failed
[root@odat1 ~]#
As you can see FAILED. Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success in the report.
The ORACHK is more tricky to check the source. To check we need go to folder /opt/oracle/dcs/oracle.ahf/orachk/SERVER/<jobID#>/orachk_<hostname><random#>/ and check the HTML file:
[root@odat1 ~]# ls -l /opt/oracle/dcs/oracle.ahf/orachk/SERVER/d7d1c155-1312-41bf-afe6-c85570073d9b/orachk_odat1_072620_14087/ total 1024 drwxr-xr-x 2 root root 4096 Jul 26 14:12 log -rw------- 1 root root 993400 Jul 26 14:12 orachk_odat1_072620_14087.html drwxr-xr-x 4 root root 24576 Jul 26 14:12 outfiles drwxr-xr-x 2 root root 4096 Jul 26 14:12 reports drwxr-xr-x 2 root root 12288 Jul 26 14:12 scripts drwxr-xr-x 2 root root 4096 Jul 26 14:12 upload [root@odat1 ~]#
There we can see internally the ERROR. In this case that we got the “Orachk validation failed” and “Software home check failed” the error is related with a check made but CVU that try to check some files (that are listed in XML file) and was not found:
Status on odat1: FAIL => Software home check failed Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libjavafx_font_t2k.so" could not be verified on node "odat1". OS error: "No such file or directory" Error Message: File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libkcms.so" could not be verified on node "odat1". OS error: "No such file or directory"
This error occurs because the CVU uses the context of file /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml to verify the files that need to be checked. But this file has more contents that need:
[root@odat1 ~]# cat /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml |grep libkcms
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
<File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
[root@odat1 ~]#
So, to fix, need to copy the file /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml over /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml. This is described at Bug 31208297 since the new version is aligned with 19c (while the other is aligned with other versions):
[root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml /tmp [root@odat1 ~]# [root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml -rw-r--r-- 1 grid oinstall 3719189 Apr 7 2019 /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml [root@odat1 ~]# [root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml -rw-r--r-- 1 grid oinstall 3723425 Apr 16 09:29 /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml [root@odat1 ~]# [root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml cp: overwrite ‘/u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml’? y [root@odat1 ~]#
Another option is simply to ignore the error and continue with the process. If you have doubts or other errors, please check the release notes from the version to check if they are listed there. Another option is open SR.
After fixing the orachk we can run the patchreport again:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.8.0.0.0
Job details
----------------------------------------------------------------
ID: e2bfcfbf-3925-40f7-b666-2e55c0d77f68
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: Created
Created: August 6, 2020 7:01:30 PM CEST
Message: Use 'odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68
Patch pre-check report
------------------------------------------------------------------------
Job ID: e2bfcfbf-3925-40f7-b666-2e55c0d77f68
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
Status: SUCCESS
Created: August 6, 2020 7:01:30 PM CEST
Result: All pre-checks succeeded
Node Name
---------------
odat1
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Patch already applied
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 31281355 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat1
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat1
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat1
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Succesfully ran Orachk
Validate command execution Success Validated command execution
Node Name
---------------
odat2
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Patch already applied
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.8.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0.
Validate clones location exist Success Validated clones location
Is patch location available Success Patch location is available.
Patch location validation Success Successfully validated location
Patch verification Success Patches 31281355 not applied on GI
home /u01/app/19.0.0.0/grid on node
odat2
Validate Opatch update Success Successfully updated the opatch in
GiHome /u01/app/19.0.0.0/grid on node
odat2
Patch conflict check Success No patch conflicts found on GiHome
/u01/app/19.0.0.0/grid on node odat2
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Succesfully ran Orachk
Validate command execution Success Validated command execution
[root@odat1 ~]#
Update-Server
After the create patchreport we can patch the server calling odacli update-server:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-server -v 19.8.0.0.0
{
"jobId" : "58472ea7-f329-45d5-a8c0-e13cf9074f5f",
"status" : "Created",
"message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
"reports" : [ ],
"createTimestamp" : "August 06, 2020 19:22:24 PM CEST",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "August 06, 2020 19:22:24 PM CEST"
}
[root@odat1 ~]#
This will create one jobIB that we can follow:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "58472ea7-f329-45d5-a8c0-e13cf9074f5f"
Job details
----------------------------------------------------------------
ID: 58472ea7-f329-45d5-a8c0-e13cf9074f5f
Description: Server Patching
Status: Success
Created: August 6, 2020 7:22:24 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation August 6, 2020 7:23:04 PM CEST August 6, 2020 7:23:05 PM CEST Success
Patch location validation August 6, 2020 7:23:04 PM CEST August 6, 2020 7:23:05 PM CEST Success
dcs-controller upgrade August 6, 2020 7:23:08 PM CEST August 6, 2020 7:23:21 PM CEST Success
dcs-controller upgrade August 6, 2020 7:23:22 PM CEST August 6, 2020 7:23:36 PM CEST Success
Patch location validation August 6, 2020 7:23:38 PM CEST August 6, 2020 7:23:38 PM CEST Success
Patch location validation August 6, 2020 7:23:38 PM CEST August 6, 2020 7:23:38 PM CEST Success
dcs-cli upgrade August 6, 2020 7:23:39 PM CEST August 6, 2020 7:23:40 PM CEST Success
dcs-cli upgrade August 6, 2020 7:23:40 PM CEST August 6, 2020 7:23:41 PM CEST Success
Creating repositories using yum August 6, 2020 7:23:46 PM CEST August 6, 2020 7:23:51 PM CEST Success
Updating YumPluginVersionLock rpm August 6, 2020 7:23:51 PM CEST August 6, 2020 7:23:51 PM CEST Success
Applying OS Patches August 6, 2020 7:23:51 PM CEST August 6, 2020 7:30:47 PM CEST Success
Creating repositories using yum August 6, 2020 7:30:48 PM CEST August 6, 2020 7:30:50 PM CEST Success
Applying HMP Patches August 6, 2020 7:30:50 PM CEST August 6, 2020 7:31:17 PM CEST Success
Patch location validation August 6, 2020 7:31:19 PM CEST August 6, 2020 7:31:19 PM CEST Success
Patch location validation August 6, 2020 7:31:19 PM CEST August 6, 2020 7:31:19 PM CEST Success
oda-hw-mgmt upgrade August 6, 2020 7:31:20 PM CEST August 6, 2020 7:31:54 PM CEST Success
oda-hw-mgmt upgrade August 6, 2020 7:31:54 PM CEST August 6, 2020 7:32:30 PM CEST Success
OSS Patching August 6, 2020 7:32:31 PM CEST August 6, 2020 7:32:32 PM CEST Success
Applying Firmware Disk Patches August 6, 2020 7:33:30 PM CEST August 6, 2020 7:34:19 PM CEST Success
Applying Firmware Expander Patches August 6, 2020 7:34:33 PM CEST August 6, 2020 7:34:50 PM CEST Success
Applying Firmware Controller Patches August 6, 2020 7:35:02 PM CEST August 6, 2020 7:35:17 PM CEST Success
Checking Ilom patch Version August 6, 2020 7:35:20 PM CEST August 6, 2020 7:35:22 PM CEST Success
Checking Ilom patch Version August 6, 2020 7:35:22 PM CEST August 6, 2020 7:35:25 PM CEST Success
Patch location validation August 6, 2020 7:35:25 PM CEST August 6, 2020 7:35:26 PM CEST Success
Patch location validation August 6, 2020 7:35:25 PM CEST August 6, 2020 7:35:26 PM CEST Success
Save password in Wallet August 6, 2020 7:35:28 PM CEST August 6, 2020 7:35:28 PM CEST Success
Apply Ilom patch August 6, 2020 7:35:28 PM CEST August 6, 2020 7:35:29 PM CEST Success
Apply Ilom patch August 6, 2020 7:35:30 PM CEST August 6, 2020 7:35:31 PM CEST Success
Copying Flash Bios to Temp location August 6, 2020 7:35:31 PM CEST August 6, 2020 7:35:31 PM CEST Success
Copying Flash Bios to Temp location August 6, 2020 7:35:31 PM CEST August 6, 2020 7:35:31 PM CEST Success
Starting the clusterware August 6, 2020 7:37:41 PM CEST August 6, 2020 7:39:40 PM CEST Success
clusterware patch verification August 6, 2020 7:39:46 PM CEST August 6, 2020 7:39:55 PM CEST Success
clusterware patch verification August 6, 2020 7:39:46 PM CEST August 6, 2020 7:39:54 PM CEST Success
Patch location validation August 6, 2020 7:39:55 PM CEST August 6, 2020 7:40:03 PM CEST Success
Patch location validation August 6, 2020 7:39:55 PM CEST August 6, 2020 7:40:03 PM CEST Success
Opatch update August 6, 2020 7:40:53 PM CEST August 6, 2020 7:40:58 PM CEST Success
Opatch update August 6, 2020 7:40:53 PM CEST August 6, 2020 7:40:56 PM CEST Success
Patch conflict check August 6, 2020 7:40:58 PM CEST August 6, 2020 7:42:09 PM CEST Success
Patch conflict check August 6, 2020 7:42:09 PM CEST August 6, 2020 7:43:22 PM CEST Success
clusterware upgrade August 6, 2020 7:43:54 PM CEST August 6, 2020 8:25:43 PM CEST Success
clusterware upgrade August 6, 2020 8:25:43 PM CEST August 6, 2020 9:08:22 PM CEST Success
Updating GiHome version August 6, 2020 9:08:22 PM CEST August 6, 2020 9:08:46 PM CEST Success
Updating GiHome version August 6, 2020 9:08:22 PM CEST August 6, 2020 9:08:46 PM CEST Success
Update System version August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:12 PM CEST Success
Update System version August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:12 PM CEST Success
preRebootNode Actions August 6, 2020 9:09:12 PM CEST August 6, 2020 9:09:58 PM CEST Success
preRebootNode Actions August 6, 2020 9:09:58 PM CEST August 6, 2020 9:10:44 PM CEST Success
Reboot Ilom August 6, 2020 9:10:44 PM CEST August 6, 2020 9:10:44 PM CEST Success
Reboot Ilom August 6, 2020 9:10:44 PM CEST August 6, 2020 9:10:45 PM CEST Success
[root@odat1 ~]#
In ODAX5-2 this process took almost two hours. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like and steps executed. Some example:
... 2020-08-06 19:43:24,793 INFO [Disabling TFA : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.z.DCSZQueue: Node not yet received from /nodes/node_1/cmd-out-q with prefix:850_2186_ 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : .---------------------------------------------------------------------------------------------. 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : +-------+---------------+-------+------+------------+----------------------+------------------+ 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | odat1 | RUNNING | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : | odat2 | RUNNING | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | 2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output : '-------+---------------+-------+------+------------+----------------------+------------------' 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.DCSProcessBuilder: Return code: 0 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.CommandExecutor: Return code: 0 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: TfaUtils: parseOutputToCheckIsTFARunning 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output: .---------------------------------------------------------------------------------------------. | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | +-------+---------------+-------+------+------------+----------------------+------------------+ | odat1 | RUNNING | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | | odat2 | RUNNING | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE | '-------+---------------+-------+------+------------+----------------------+------------------' 2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output by line: .---------------------------------------------------------------------------------------------. ...
After you finish this jobIB, and around 10 minutes later, ODA nodes will reboot to boot the new version.
Update-storage
The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not update the firmware of the disks as an example.
You can see from odacli describe-component that some part are not up to date:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.8.0.0.0
System node Name
---------------
odat1
Local System Version
---------------
19.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.8.0.0.0 up-to-date
GI 19.8.0.0.200714 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.11.0.0.200714
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200714
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200714
}
DCSAGENT 19.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 16.00.08.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
System node Name
---------------
odat2
Local System Version
---------------
19.8.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.8.0.0.0 up-to-date
GI 19.8.0.0.200714 up-to-date
DB {
[ OraDB18000_home1 ] 18.3.0.0.180717 18.11.0.0.200714
[ OraDB12102_home1,OraDB12102_home2 ] 12.1.0.2.180717 12.1.0.2.200714
[ OraDB11204_home1,OraDB11204_home2 ] 11.2.0.4.180717 11.2.0.4.200714
}
DCSAGENT 19.8.0.0.0 up-to-date
ILOM 4.0.4.52.r132805 up-to-date
BIOS 30300200 up-to-date
OS 7.8 up-to-date
FIRMWARECONTROLLER {
[ c0 ] 4.650.00-7176 up-to-date
[ c1,c2 ] 16.00.08.00 up-to-date
}
FIRMWAREEXPANDER 001E up-to-date
FIRMWAREDISK {
[ c0d0,c0d1 ] A7E0 up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6, PD51 up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21, A29A up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}
HMP 2.4.5.0.1 up-to-date
[root@odat1 ~]#
Even if everything is up-to-date we can call odacli update-storage to patch everything:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.8.0.0.0
{
"jobId" : "cf7e78b9-c476-4f65-964a-0126e9a4d1f3",
"status" : "Created",
"message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "August 06, 2020 21:27:49 PM CEST",
"resourceList" : [ ],
"description" : "Storage Firmware Patching",
"updatedTime" : "August 06, 2020 21:27:49 PM CEST"
}
[root@odat1 ~]#
And we can follow the jobID:
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "cf7e78b9-c476-4f65-964a-0126e9a4d1f3"
Job details
----------------------------------------------------------------
ID: cf7e78b9-c476-4f65-964a-0126e9a4d1f3
Description: Storage Firmware Patching
Status: Success
Created: August 6, 2020 9:27:49 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches August 6, 2020 9:32:08 PM CEST August 6, 2020 9:34:38 PM CEST Success
Applying Firmware Controller Patches August 6, 2020 9:34:52 PM CEST August 6, 2020 9:35:07 PM CEST Success
preRebootNode Actions August 6, 2020 9:35:08 PM CEST August 6, 2020 9:35:08 PM CEST Success
preRebootNode Actions August 6, 2020 9:35:08 PM CEST August 6, 2020 9:35:08 PM CEST Success
Reboot Ilom August 6, 2020 9:35:09 PM CEST August 6, 2020 9:35:09 PM CEST Success
Reboot Ilom August 6, 2020 9:35:09 PM CEST August 6, 2020 9:35:09 PM CEST Success
[root@odat1 ~]#
After it finishes, the ODA nodes will reboot again. To reboot be effective maybe you need to wait 10 minutes or more.
And after the reboot, you can see that everything is up-to-date using odacli update-storage.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: Upgrade AHF and TFA at ODA - Fernando Simon