Patch ODA from 18.3 to 19.8. Part 4 – 19.7 to 19.8

The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.

The fourth part covers the upgrade from 19.7 to 19.8. I separate so you can use this as a direct guide from 19.6 to 19.7 if you need to do just this update. Parts of this post are similar to the upgrade from 19.6 to 19.7 that I described in the previous post.

Patch Matrix

The matrix of what can be done can be found at this post from ODA blog, and you can check below:

Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.

Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.

Actual environment

The environment is ODA X5-2, running the version 19.7. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it is running bare metal (BM).

The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.

I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):

[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/
cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles'
cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig'
[root@odat1 ~]#
[root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/
[root@odat1 ~]#

Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component. If we need we just need to update the components with odacli update-storage. This will (if some firmware is applied) reboot the ODA.

Before starting the patch I recommend stopping all databases running at ODA. This will speed up the process. So, left just GI and ASM running.

19.8

The upgrade from 19.7 to 19.8 is a simple process and follows a well-known procedure if you compare with the upgrade of 18.8 to 19.6. The 19.8 version enables some features, like OVM. But here I am doing the patch over BM and will continue to use this here. The patch to download is 31481816.

Update repository

The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):

[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli update-repository -f /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip
{
  "jobId" : "0c4b1959-0f9c-4a41-9824-f7878a13719d",
  "status" : "Created",
  "message" : "/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip",
  "reports" : [ ],
  "createTimestamp" : "July 29, 2020 17:33:52 PM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "July 29, 2020 17:33:52 PM CEST"
}
[root@odat1 19.8]#

After that we can follow the job with odacli describe-job:

[root@odat1 19.8]# /opt/oracle/dcs/bin/odacli describe-job -i "0c4b1959-0f9c-4a41-9824-f7878a13719d"

Job details
----------------------------------------------------------------
                     ID:  0c4b1959-0f9c-4a41-9824-f7878a13719d
            Description:  Repository Update
                 Status:  Success
                Created:  July 29, 2020 5:33:52 PM CEST
                Message:  /NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server1of2.zip,/NAS/ZONE_22/ODA_PATCH_18/19.8/oda-sm-19.8.0.0.0-200718-server2of2.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     July 29, 2020 5:33:53 PM CEST       July 29, 2020 5:33:53 PM CEST       Success
Setting up ssh equivalance               July 29, 2020 5:33:54 PM CEST       July 29, 2020 5:33:54 PM CEST       Success
Copy BundleFile                          July 29, 2020 5:33:55 PM CEST       July 29, 2020 5:38:07 PM CEST       Success
Validating CopiedFile                    July 29, 2020 5:38:07 PM CEST       July 29, 2020 5:38:41 PM CEST       Success
Unzip bundle                             July 29, 2020 5:38:41 PM CEST       July 29, 2020 5:56:35 PM CEST       Success
Unzip bundle                             July 29, 2020 5:56:35 PM CEST       July 29, 2020 5:58:36 PM CEST       Success
Delete PatchBundles                      July 29, 2020 5:58:43 PM CEST       July 29, 2020 5:58:58 PM CEST       Success
Removing ssh keys                        July 29, 2020 5:58:59 PM CEST       July 29, 2020 5:59:00 PM CEST       Success

[root@odat1 19.8]#

DCSAgent

The next step is to update the internal DCS agent for ODA 19.8. Is a simple step executed using the odacli update-dcsagent. The parameter is the version 19.8.0.0:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.8.0.0.0
{
  "jobId" : "a3ad9edf-f58d-40b0-9ea0-75e9347907c3",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "August 06, 2020 17:30:46 PM CEST",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "August 06, 2020 17:30:46 PM CEST"
}
[root@odat1 ~]#

And as usual, we need to wait for the job to finish before continuing. We use the odacli describe-job to check this:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3"

Job details
----------------------------------------------------------------
                     ID:  a3ad9edf-f58d-40b0-9ea0-75e9347907c3
            Description:  DcsAgent patching
                 Status:  Success
                Created:  August 6, 2020 5:30:46 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade  to version 19.8.0.0.0 August 6, 2020 5:30:47 PM CEST      August 6, 2020 5:32:59 PM CEST      Success
dcs-agent upgrade  to version 19.8.0.0.0 August 6, 2020 5:32:59 PM CEST      August 6, 2020 5:34:54 PM CEST      Success
Update System version                    August 6, 2020 5:35:00 PM CEST      August 6, 2020 5:35:00 PM CEST      Success
Update System version                    August 6, 2020 5:35:01 PM CEST      August 6, 2020 5:35:01 PM CEST      Success

[root@odat1 ~]#

You can receive some errors if you try to describe the jobID. This can occur because the process restarts the DCS. If you got this, just wait some minutes and try again (some common errors are: DCS-10001:Internal error encountered: Fail to get credential or DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "a3ad9edf-f58d-40b0-9ea0-75e9347907c3"
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
[root@odat1 ~]#

DCSAdmin

The next step will be to update the dcsadmin using the command odacli update-dcsadmin:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.7.0.0.0
{
  "jobId" : "edf88f79-529c-438e-8a4a-a412749c8a70",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 11:56:45 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "July 26, 2020 11:56:45 AM CEST"
}
[root@odat1 ~]#

And as usual, we can check the end of the job before continue:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "e00b6dd8-d315-40de-b612-641d4853fe05"

Job details
----------------------------------------------------------------
                     ID:  e00b6dd8-d315-40de-b612-641d4853fe05
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  August 6, 2020 5:36:09 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                August 6, 2020 5:36:12 PM CEST      August 6, 2020 5:36:12 PM CEST      Success
Patch location validation                August 6, 2020 5:36:12 PM CEST      August 6, 2020 5:36:12 PM CEST      Success
dcsadmin upgrade                         August 6, 2020 5:36:13 PM CEST      August 6, 2020 5:36:21 PM CEST      Success
dcsadmin upgrade                         August 6, 2020 5:36:21 PM CEST      August 6, 2020 5:36:29 PM CEST      Success
Update System version                    August 6, 2020 5:36:30 PM CEST      August 6, 2020 5:36:30 PM CEST      Success
Update System version                    August 6, 2020 5:36:30 PM CEST      August 6, 2020 5:36:30 PM CEST      Success

[root@odat1 ~]#

DCSComponents

The next is to update the dcscomponents with command odacli update-dcscomponents. As occurred with other versions, the commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.8.0.0.0
{
  "jobId" : "0f27ebd9-591f-4242-9e89-b998d93ecede",
  "status" : "Success",
  "message" : null,
  "reports" : null,
  "createTimestamp" : "August 06, 2020 17:37:01 PM CEST",
  "description" : "Job completed and is not part of Agent job list",
  "updatedTime" : "August 06, 2020 17:37:01 PM CEST"
} 
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "0f27ebd9-591f-4242-9e89-b998d93ecede"
DCS-10000:Resource Job with ID 0f27ebd9-591f-4242-9e89-b998d93ecede is not found.
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
...
0c4b1959-0f9c-4a41-9824-f7878a13719d     Repository Update                                                           July 29, 2020 5:33:52 PM CEST       Success
a3ad9edf-f58d-40b0-9ea0-75e9347907c3     DcsAgent patching                                                           August 6, 2020 5:30:46 PM CEST      Success
e00b6dd8-d315-40de-b612-641d4853fe05     DcsAdmin patching                                                           August 6, 2020 5:36:09 PM CEST      Success
cae0726b-3298-4c02-ba8e-b4be76eab620     SSH keys update                                                             August 6, 2020 5:37:24 PM CEST      Success
a702447e-71ed-42af-904c-1e3225f193df     SSH key delete                                                              August 6, 2020 5:37:29 PM CEST      Success
348599c5-2718-40fa-8bbf-f3ecfee28123     SSH keys update                                                             August 6, 2020 5:37:59 PM CEST      Success
cff63e2a-be64-4e2d-9500-f274d6c2a640     SSH key delete                                                              August 6, 2020 5:39:33 PM CEST      Success

[root@odat1 ~]#

Create-prepatchreport

The next step is to verify if you can update it to 19.8. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.7.0.0.0

Job details
----------------------------------------------------------------
                     ID:  d7d1c155-1312-41bf-afe6-c85570073d9b
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  July 26, 2020 1:59:04 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odat1 ~]#

And after some time we got the full result (30 minutes more or less):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e150f6d2-19c2-40c9-90da-3d427234d1b6

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  e150f6d2-19c2-40c9-90da-3d427234d1b6
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  August 6, 2020 6:28:15 PM CEST
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
odat1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.8.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 31281355 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat1
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat1
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat1
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Software home                   Failed    Software home check failed

Node Name
---------------
odat2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.8.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 31281355 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat2
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat2
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat2
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Software home                   Failed    Software home check failed



[root@odat1 ~]#

As you can see FAILED. Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success in the report.

The ORACHK is more tricky to check the source. To check we need go to folder /opt/oracle/dcs/oracle.ahf/orachk/SERVER/<jobID#>/orachk_<hostname><random#>/ and check the HTML file:

[root@odat1 ~]# ls -l /opt/oracle/dcs/oracle.ahf/orachk/SERVER/d7d1c155-1312-41bf-afe6-c85570073d9b/orachk_odat1_072620_14087/
total 1024
drwxr-xr-x 2 root root   4096 Jul 26 14:12 log
-rw------- 1 root root 993400 Jul 26 14:12 orachk_odat1_072620_14087.html
drwxr-xr-x 4 root root  24576 Jul 26 14:12 outfiles
drwxr-xr-x 2 root root   4096 Jul 26 14:12 reports
drwxr-xr-x 2 root root  12288 Jul 26 14:12 scripts
drwxr-xr-x 2 root root   4096 Jul 26 14:12 upload
[root@odat1 ~]#

There we can see internally the ERROR. In this case that we got the “Orachk validation failed” and “Software home check failed” the error is related with a check made but CVU that try to check some files (that are listed in XML file) and was not found:

Status on odat1:
FAIL => Software home check failed


Error Message:
File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libjavafx_font_t2k.so" could not be verified on node "odat1". OS error: "No such file or directory"
Error Message:
File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libkcms.so" could not be verified on node "odat1". OS error: "No such file or directory"

This error occurs because the CVU uses the context of file /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml to verify the files that need to be checked. But this file has more contents that need:

[root@odat1 ~]# cat /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml |grep libkcms
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
[root@odat1 ~]#

So, to fix, need to copy the file /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml over /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml. This is described at Bug 31208297 since the new version is aligned with 19c (while the other is aligned with other versions):

[root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml /tmp
[root@odat1 ~]#
[root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml
-rw-r--r-- 1 grid oinstall 3719189 Apr  7  2019 /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml
[root@odat1 ~]#
[root@odat1 ~]# ls -l /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml
-rw-r--r-- 1 grid oinstall 3723425 Apr 16 09:29 /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml
[root@odat1 ~]#
[root@odat1 ~]# cp /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml
cp: overwrite ‘/u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml’? y
[root@odat1 ~]#

Another option is simply to ignore the error and continue with the process. If you have doubts or other errors, please check the release notes from the version to check if they are listed there. Another option is open SR.

After fixing the orachk we can run the patchreport again:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.8.0.0.0

Job details
----------------------------------------------------------------
                     ID:  e2bfcfbf-3925-40f7-b666-2e55c0d77f68
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  August 6, 2020 7:01:30 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i e2bfcfbf-3925-40f7-b666-2e55c0d77f68

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  e2bfcfbf-3925-40f7-b666-2e55c0d77f68
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  August 6, 2020 7:01:30 PM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
odat1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.8.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 31281355 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat1
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat1
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat1
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Succesfully ran Orachk
Validate command execution      Success   Validated command execution

Node Name
---------------
odat2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.8.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.8.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 31281355 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat2
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat2
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat2
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Succesfully ran Orachk
Validate command execution      Success   Validated command execution



[root@odat1 ~]#

Update-Server

After the create patchreport we can patch the server calling odacli update-server:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-server -v 19.8.0.0.0
{
  "jobId" : "58472ea7-f329-45d5-a8c0-e13cf9074f5f",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "August 06, 2020 19:22:24 PM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "August 06, 2020 19:22:24 PM CEST"
}
[root@odat1 ~]#

This will create one jobIB that we can follow:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "58472ea7-f329-45d5-a8c0-e13cf9074f5f"

Job details
----------------------------------------------------------------
                     ID:  58472ea7-f329-45d5-a8c0-e13cf9074f5f
            Description:  Server Patching
                 Status:  Success
                Created:  August 6, 2020 7:22:24 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                August 6, 2020 7:23:04 PM CEST      August 6, 2020 7:23:05 PM CEST      Success
Patch location validation                August 6, 2020 7:23:04 PM CEST      August 6, 2020 7:23:05 PM CEST      Success
dcs-controller upgrade                   August 6, 2020 7:23:08 PM CEST      August 6, 2020 7:23:21 PM CEST      Success
dcs-controller upgrade                   August 6, 2020 7:23:22 PM CEST      August 6, 2020 7:23:36 PM CEST      Success
Patch location validation                August 6, 2020 7:23:38 PM CEST      August 6, 2020 7:23:38 PM CEST      Success
Patch location validation                August 6, 2020 7:23:38 PM CEST      August 6, 2020 7:23:38 PM CEST      Success
dcs-cli upgrade                          August 6, 2020 7:23:39 PM CEST      August 6, 2020 7:23:40 PM CEST      Success
dcs-cli upgrade                          August 6, 2020 7:23:40 PM CEST      August 6, 2020 7:23:41 PM CEST      Success
Creating repositories using yum          August 6, 2020 7:23:46 PM CEST      August 6, 2020 7:23:51 PM CEST      Success
Updating YumPluginVersionLock rpm        August 6, 2020 7:23:51 PM CEST      August 6, 2020 7:23:51 PM CEST      Success
Applying OS Patches                      August 6, 2020 7:23:51 PM CEST      August 6, 2020 7:30:47 PM CEST      Success
Creating repositories using yum          August 6, 2020 7:30:48 PM CEST      August 6, 2020 7:30:50 PM CEST      Success
Applying HMP Patches                     August 6, 2020 7:30:50 PM CEST      August 6, 2020 7:31:17 PM CEST      Success
Patch location validation                August 6, 2020 7:31:19 PM CEST      August 6, 2020 7:31:19 PM CEST      Success
Patch location validation                August 6, 2020 7:31:19 PM CEST      August 6, 2020 7:31:19 PM CEST      Success
oda-hw-mgmt upgrade                      August 6, 2020 7:31:20 PM CEST      August 6, 2020 7:31:54 PM CEST      Success
oda-hw-mgmt upgrade                      August 6, 2020 7:31:54 PM CEST      August 6, 2020 7:32:30 PM CEST      Success
OSS Patching                             August 6, 2020 7:32:31 PM CEST      August 6, 2020 7:32:32 PM CEST      Success
Applying Firmware Disk Patches           August 6, 2020 7:33:30 PM CEST      August 6, 2020 7:34:19 PM CEST      Success
Applying Firmware Expander Patches       August 6, 2020 7:34:33 PM CEST      August 6, 2020 7:34:50 PM CEST      Success
Applying Firmware Controller Patches     August 6, 2020 7:35:02 PM CEST      August 6, 2020 7:35:17 PM CEST      Success
Checking Ilom patch Version              August 6, 2020 7:35:20 PM CEST      August 6, 2020 7:35:22 PM CEST      Success
Checking Ilom patch Version              August 6, 2020 7:35:22 PM CEST      August 6, 2020 7:35:25 PM CEST      Success
Patch location validation                August 6, 2020 7:35:25 PM CEST      August 6, 2020 7:35:26 PM CEST      Success
Patch location validation                August 6, 2020 7:35:25 PM CEST      August 6, 2020 7:35:26 PM CEST      Success
Save password in Wallet                  August 6, 2020 7:35:28 PM CEST      August 6, 2020 7:35:28 PM CEST      Success
Apply Ilom patch                         August 6, 2020 7:35:28 PM CEST      August 6, 2020 7:35:29 PM CEST      Success
Apply Ilom patch                         August 6, 2020 7:35:30 PM CEST      August 6, 2020 7:35:31 PM CEST      Success
Copying Flash Bios to Temp location      August 6, 2020 7:35:31 PM CEST      August 6, 2020 7:35:31 PM CEST      Success
Copying Flash Bios to Temp location      August 6, 2020 7:35:31 PM CEST      August 6, 2020 7:35:31 PM CEST      Success
Starting the clusterware                 August 6, 2020 7:37:41 PM CEST      August 6, 2020 7:39:40 PM CEST      Success
clusterware patch verification           August 6, 2020 7:39:46 PM CEST      August 6, 2020 7:39:55 PM CEST      Success
clusterware patch verification           August 6, 2020 7:39:46 PM CEST      August 6, 2020 7:39:54 PM CEST      Success
Patch location validation                August 6, 2020 7:39:55 PM CEST      August 6, 2020 7:40:03 PM CEST      Success
Patch location validation                August 6, 2020 7:39:55 PM CEST      August 6, 2020 7:40:03 PM CEST      Success
Opatch update                            August 6, 2020 7:40:53 PM CEST      August 6, 2020 7:40:58 PM CEST      Success
Opatch update                            August 6, 2020 7:40:53 PM CEST      August 6, 2020 7:40:56 PM CEST      Success
Patch conflict check                     August 6, 2020 7:40:58 PM CEST      August 6, 2020 7:42:09 PM CEST      Success
Patch conflict check                     August 6, 2020 7:42:09 PM CEST      August 6, 2020 7:43:22 PM CEST      Success
clusterware upgrade                      August 6, 2020 7:43:54 PM CEST      August 6, 2020 8:25:43 PM CEST      Success
clusterware upgrade                      August 6, 2020 8:25:43 PM CEST      August 6, 2020 9:08:22 PM CEST      Success
Updating GiHome version                  August 6, 2020 9:08:22 PM CEST      August 6, 2020 9:08:46 PM CEST      Success
Updating GiHome version                  August 6, 2020 9:08:22 PM CEST      August 6, 2020 9:08:46 PM CEST      Success
Update System version                    August 6, 2020 9:09:12 PM CEST      August 6, 2020 9:09:12 PM CEST      Success
Update System version                    August 6, 2020 9:09:12 PM CEST      August 6, 2020 9:09:12 PM CEST      Success
preRebootNode Actions                    August 6, 2020 9:09:12 PM CEST      August 6, 2020 9:09:58 PM CEST      Success
preRebootNode Actions                    August 6, 2020 9:09:58 PM CEST      August 6, 2020 9:10:44 PM CEST      Success
Reboot Ilom                              August 6, 2020 9:10:44 PM CEST      August 6, 2020 9:10:44 PM CEST      Success
Reboot Ilom                              August 6, 2020 9:10:44 PM CEST      August 6, 2020 9:10:45 PM CEST      Success

[root@odat1 ~]#

In ODAX5-2 this process took almost two hours. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like and steps executed. Some example:

...
2020-08-06 19:43:24,793 INFO [Disabling TFA : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.z.DCSZQueue: Node not yet received from /nodes/node_1/cmd-out-q with prefix:850_2186_
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
.---------------------------------------------------------------------------------------------.
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
| Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status |
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
+-------+---------------+-------+------+------------+----------------------+------------------+
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
| odat1 | RUNNING | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE |
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
| odat2 | RUNNING | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE |
2020-08-06 19:43:25,845 DEBUG [Thread-665] [] c.o.d.c.u.CommonsUtils: Output :
'-------+---------------+-------+------+------------+----------------------+------------------'
2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.DCSProcessBuilder: Return code: 0
2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.c.u.c.CommandExecutor: Return code: 0
2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: TfaUtils: parseOutputToCheckIsTFARunning
2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output: .---------------------------------------------------------------------------------------------.
| Host  | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+-------+---------------+-------+------+------------+----------------------+------------------+
| odat1 | RUNNING       | 51954 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE         |
| odat2 | RUNNING       | 88162 | 5000 | 20.2.1.0.0 | 20210020200706165623 | COMPLETE         |
'-------+---------------+-------+------+------------+----------------------+------------------'
2020-08-06 19:43:25,877 DEBUG [Server patching : JobId=58472ea7-f329-45d5-a8c0-e13cf9074f5f] [] c.o.d.a.r.s.t.TfaUtils: tfactl error output by line: .---------------------------------------------------------------------------------------------.
...

After you finish this jobIB, and around 10 minutes later, ODA nodes will reboot to boot the new version.

Update-storage

The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not update the firmware of the disks as an example.

You can see from odacli describe-component that some part are not up to date:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.8.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
19.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.8.0.0.0            up-to-date

GI                                        19.8.0.0.200714       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.11.0.0.200714
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200714
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200714
}

DCSAGENT                                  19.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 16.00.08.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date

System node Name
---------------
odat2

Local System Version
---------------
19.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.8.0.0.0            up-to-date

GI                                        19.8.0.0.200714       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.11.0.0.200714
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200714
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200714
}

DCSAGENT                                  19.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 16.00.08.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date


[root@odat1 ~]#

Even if everything is up-to-date we can call odacli update-storage to patch everything:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.8.0.0.0
{
  "jobId" : "cf7e78b9-c476-4f65-964a-0126e9a4d1f3",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "August 06, 2020 21:27:49 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "August 06, 2020 21:27:49 PM CEST"
}
[root@odat1 ~]#

And we can follow the jobID:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "cf7e78b9-c476-4f65-964a-0126e9a4d1f3"

Job details
----------------------------------------------------------------
                     ID:  cf7e78b9-c476-4f65-964a-0126e9a4d1f3
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  August 6, 2020 9:27:49 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           August 6, 2020 9:32:08 PM CEST      August 6, 2020 9:34:38 PM CEST      Success
Applying Firmware Controller Patches     August 6, 2020 9:34:52 PM CEST      August 6, 2020 9:35:07 PM CEST      Success
preRebootNode Actions                    August 6, 2020 9:35:08 PM CEST      August 6, 2020 9:35:08 PM CEST      Success
preRebootNode Actions                    August 6, 2020 9:35:08 PM CEST      August 6, 2020 9:35:08 PM CEST      Success
Reboot Ilom                              August 6, 2020 9:35:09 PM CEST      August 6, 2020 9:35:09 PM CEST      Success
Reboot Ilom                              August 6, 2020 9:35:09 PM CEST      August 6, 2020 9:35:09 PM CEST      Success

[root@odat1 ~]#

After it finishes, the ODA nodes will reboot again. To reboot be effective maybe you need to wait 10 minutes or more.

And after the reboot, you can see that everything is up-to-date using odacli update-storage.

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

Leave a Reply

Your email address will not be published. Required fields are marked *