Patch ODA from 18.3 to 19.8. Part 3 – 19.6 to 19.7

This is the third part of the ODA patch series, from 18.3 to 19.8. I separate in multiple parts and you can use this part as a direct guide to patch ODA from 19.6 to 19.7. Each part can be used alone since they cover all the needed steps. Some steps of this post are similar to the upgrade from 18.8 to 19.6 that I described in the previous post.

The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version. 

Patch Matrix

The matrix of what can be done can be found at this post from ODA blog, and you can check below:

Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.

Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.

Actual environment

The environment is ODA X5-2, running the version 19.6. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it running bare metal (BM).

The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.

I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):

[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/
cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles'
cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig'
[root@odat1 ~]#
[root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/
[root@odat1 ~]#

Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component. If we need we just need to update the components with odacli update-storage. This will (if some firmware is applied) reboot the ODA.

Before starting the patch I recommend stopping all databases running at ODA. This will speed up the process. So, left just GI and ASM running.

If you got any error in the next steps related to lack of space. You can check my previous posts of this series because there I describe how to increase the filesystem if needed. And I also show how to decrease the filesystem as well (like /u01) to have raw space that can be allocated to other mountpoints.

19.7

The upgrade from 19.6 to 19.7 is a simple process and follows a well know procedure. In this process, we do not upgrade Linux (like 18.8 to 19.6), just some new rpms and we upgrade the GI to 19.6. The patch to download is 31220620.

Update repository

The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-repository -f /nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip
{
  "jobId" : "ad0389f3-2e2b-4523-b054-de8e84a84949",
  "status" : "Created",
  "message" : "/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip",
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 11:21:27 AM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "July 26, 2020 11:21:27 AM CEST"
}
You have new mail in /var/spool/mail/root
[root@odat1 ~]#

After that we can follow the job with odacli describe-job:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "ad0389f3-2e2b-4523-b054-de8e84a84949"

Job details
----------------------------------------------------------------
                     ID:  ad0389f3-2e2b-4523-b054-de8e84a84949
            Description:  Repository Update
                 Status:  Success
                Created:  July 26, 2020 11:21:27 AM CEST
                Message:  /nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server1of2.zip,/nfs/ODA_PATCH_18/19.7/oda-sm-19.7.0.0.0-200520-server2of2.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     July 26, 2020 11:21:28 AM CEST      July 26, 2020 11:21:28 AM CEST      Success
Setting up ssh equivalance               July 26, 2020 11:21:29 AM CEST      July 26, 2020 11:21:29 AM CEST      Success
Copy BundleFile                          July 26, 2020 11:21:30 AM CEST      July 26, 2020 11:25:50 AM CEST      Success
Validating CopiedFile                    July 26, 2020 11:25:50 AM CEST      July 26, 2020 11:26:21 AM CEST      Success
Unzip bundle                             July 26, 2020 11:26:22 AM CEST      July 26, 2020 11:43:55 AM CEST      Success
Unzip bundle                             July 26, 2020 11:43:55 AM CEST      July 26, 2020 11:46:16 AM CEST      Success
Delete PatchBundles                      July 26, 2020 11:46:22 AM CEST      July 26, 2020 11:46:45 AM CEST      Success
Removing ssh keys                        July 26, 2020 11:46:45 AM CEST      July 26, 2020 11:46:47 AM CEST      Success

[root@odat1 ~]#

DCSAgent

The next step is to update the internal DCS agent for ODA 19.7. Is a simple step executed using the odacli update-dcsagent. The parameter is the version 19.7.0.0:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.7.0.0.0
{
  "jobId" : "1af29269-73cd-4eb6-8208-e5885376b274",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 11:51:01 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "July 26, 2020 11:51:01 AM CEST"
}
[root@odat1 ~]#

And as usual, we need to wait for the job to finish before continuing. We use the odacli describe-job to check this:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "1af29269-73cd-4eb6-8208-e5885376b274"

Job details
----------------------------------------------------------------
                     ID:  1af29269-73cd-4eb6-8208-e5885376b274
            Description:  DcsAgent patching
                 Status:  Success
                Created:  July 26, 2020 11:51:01 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade  to version 19.7.0.0.0 July 26, 2020 11:51:02 AM CEST      July 26, 2020 11:52:49 AM CEST      Success
dcs-agent upgrade  to version 19.7.0.0.0 July 26, 2020 11:52:49 AM CEST      July 26, 2020 11:54:28 AM CEST      Success
Update System version                    July 26, 2020 11:54:35 AM CEST      July 26, 2020 11:54:35 AM CEST      Success
Update System version                    July 26, 2020 11:54:35 AM CEST      July 26, 2020 11:54:35 AM CEST      Success

[root@odat1 ~]#

You can receive some errors if you try to describe the jobID. This can occur because the process restarts the DCS. If you got this, just wait some minutes and try again (some common errors are: DCS-10001:Internal error encountered: Fail to get credential or DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070).

DCSAdmin

The next step will be to update the dcsadmin using the command odacli update-dcsadmin:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.7.0.0.0
{
  "jobId" : "edf88f79-529c-438e-8a4a-a412749c8a70",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 11:56:45 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "July 26, 2020 11:56:45 AM CEST"
}
[root@odat1 ~]#

And as usual, we can check the end of the job before continue:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "edf88f79-529c-438e-8a4a-a412749c8a70"

Job details
----------------------------------------------------------------
                     ID:  edf88f79-529c-438e-8a4a-a412749c8a70
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  July 26, 2020 11:56:45 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 26, 2020 11:56:47 AM CEST      July 26, 2020 11:56:47 AM CEST      Success
Patch location validation                July 26, 2020 11:56:47 AM CEST      July 26, 2020 11:56:47 AM CEST      Success
dcsadmin upgrade                         July 26, 2020 11:56:48 AM CEST      July 26, 2020 11:56:54 AM CEST      Success
dcsadmin upgrade                         July 26, 2020 11:56:54 AM CEST      July 26, 2020 11:57:01 AM CEST      Success
Update System version                    July 26, 2020 11:57:02 AM CEST      July 26, 2020 11:57:02 AM CEST      Success
Update System version                    July 26, 2020 11:57:02 AM CEST      July 26, 2020 11:57:02 AM CEST      Success

[root@odat1 ~]#

DCSComponents

The next is to update the dcscomponents with command odacli update-dcscomponents. As occurred with other versions, the commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.7.0.0.0
{
  "jobId" : "e4f87d5e-9b79-4bcd-a74f-daa9ff672474",
  "status" : "Success",
  "message" : null,
  "reports" : null,
  "createTimestamp" : "July 26, 2020 11:57:28 AM CEST",
  "description" : "Job completed and is not part of Agent job list",
  "updatedTime" : "July 26, 2020 11:57:28 AM CEST"
}
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "e4f87d5e-9b79-4bcd-a74f-daa9ff672474"
DCS-10000:Resource Job with ID e4f87d5e-9b79-4bcd-a74f-daa9ff672474 is not found.
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
…
…
ad0389f3-2e2b-4523-b054-de8e84a84949     Repository Update                                                           July 26, 2020 11:21:27 AM CEST      Success
1af29269-73cd-4eb6-8208-e5885376b274     DcsAgent patching                                                           July 26, 2020 11:51:01 AM CEST      Success
edf88f79-529c-438e-8a4a-a412749c8a70     DcsAdmin patching                                                           July 26, 2020 11:56:45 AM CEST      Success
c8d8bcde-4015-468d-8332-d2569e5ffa6d     SSH keys update                                                             July 26, 2020 11:57:51 AM CEST      Success
a1c4cdc8-1b9e-4d25-8fac-cf85ae561e5e     SSH key delete                                                              July 26, 2020 11:57:56 AM CEST      Success
7a9aff35-a04e-43ab-bc3b-7c3aee9a72d9     SSH keys update                                                             July 26, 2020 11:58:25 AM CEST      Success
d0817c75-3fa3-4974-ade1-470a0c7993d8     SSH key delete                                                              July 26, 2020 12:00:00 PM CEST      Success

[root@odat1 ~]#

Create-prepatchreport

The next step is to verify if you can update it to 19.7. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.7.0.0.0

Job details
----------------------------------------------------------------
                     ID:  d7d1c155-1312-41bf-afe6-c85570073d9b
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  July 26, 2020 1:59:04 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odat1 ~]#

And after some time we got the full result (30 minutes more or less):

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i d7d1c155-1312-41bf-afe6-c85570073d9b

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d7d1c155-1312-41bf-afe6-c85570073d9b
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  July 26, 2020 1:59:04 PM CEST
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
odat1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.7.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 30869156 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat1
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat1
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat1

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Software home                   Failed    Software home check failed

Node Name
---------------
odat2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.7.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.7.0.0.0.
Validate clones location exist  Success   Validated clones location
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 30869156 not applied on GI
                                          home /u01/app/19.0.0.0/grid on node
                                          odat2
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/grid on node
                                          odat2
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/grid on node odat2

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Software home                   Failed    Software home check failed



[root@odat1 ~]#

As you can see FAILED. Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success in the report.

The ORACHK is more tricky to check the source. To check we need go to folder /opt/oracle/dcs/oracle.ahf/orachk/SERVER/<jobID#>/orachk_<hostname><random#>/ and check the HTML file:

[root@odat1 ~]# ls -l /opt/oracle/dcs/oracle.ahf/orachk/SERVER/d7d1c155-1312-41bf-afe6-c85570073d9b/orachk_odat1_072620_14087/
total 1024
drwxr-xr-x 2 root root   4096 Jul 26 14:12 log
-rw------- 1 root root 993400 Jul 26 14:12 orachk_odat1_072620_14087.html
drwxr-xr-x 4 root root  24576 Jul 26 14:12 outfiles
drwxr-xr-x 2 root root   4096 Jul 26 14:12 reports
drwxr-xr-x 2 root root  12288 Jul 26 14:12 scripts
drwxr-xr-x 2 root root   4096 Jul 26 14:12 upload
[root@odat1 ~]#

There we can see internally the ERROR. In this case that we got the “Orachk validation failed” and “Software home check failed” the error is related with a check made but CVU that try to check some files (that are listed in XML file) and was not found:

Cluster Verification Utility (CVU 19.6.0.0.0 ) result
Status	Type	Message	Status On	Details
FAIL	OS Check	Software home check failed	All Database Servers	
View

Status on odat1:
FAIL => Software home check failed


Error Message:
File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libjavafx_font_t2k.so" could not be verified on node "odat1". OS error: "No such file or directory"
Error Message:
File "/u01/app/19.0.0.0/grid/jdk/jre/lib/amd64/libkcms.so" could not be verified on node "odat1". OS error: "No such file or directory"
Error Message:
File "/u01/app/19.0.0.0/grid/rdbms/lib/ksms.o" could not be verified on node "odat1". OS error: "No such file or directory"

This error occurs because the CVU uses the context of file /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml to verify the files that need to be checked. But this file has more contents that need:

[root@odat1 ~]# cat /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml |grep libkcms
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
         <File Path="jdk/jre/lib/amd64/" Name="libkcms.so"/>
[root@odat1 ~]#

So, to fix, need to copy the file /u01/app/19.0.0.0/grid/cv/cvdata/19/ora_software_cfg.xml over /u01/app/19.0.0.0/grid/cv/cvdata/ora_software_cfg.xml. This is described at Bug 31208297 since the new version is aligned with 19c (while the other is aligned with other versions).

Another option is simply ignore the error and continue with the process. If you have doubts or other errors, please check the release notes from the version to check if they are listed there. Another option is open SR.

Update-Server

After the create patchreport we can patch the server calling odacli update-server:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-server -v 19.7.0.0.0
{
  "jobId" : "91abf816-b171-4e2f-a206-23013f53fb8e",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 16:04:06 PM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "July 26, 2020 16:04:06 PM CEST"
}
[root@odat1 ~]#

This will create one jobIB that we can follow:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "91abf816-b171-4e2f-a206-23013f53fb8e"

Job details
----------------------------------------------------------------
                     ID:  91abf816-b171-4e2f-a206-23013f53fb8e
            Description:  Server Patching
                 Status:  Success
                Created:  July 26, 2020 4:04:06 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 26, 2020 4:04:42 PM CEST       July 26, 2020 4:04:43 PM CEST       Success
Patch location validation                July 26, 2020 4:04:42 PM CEST       July 26, 2020 4:04:42 PM CEST       Success
dcs-controller upgrade                   July 26, 2020 4:04:43 PM CEST       July 26, 2020 4:04:50 PM CEST       Success
dcs-controller upgrade                   July 26, 2020 4:04:50 PM CEST       July 26, 2020 4:04:57 PM CEST       Success
Patch location validation                July 26, 2020 4:05:00 PM CEST       July 26, 2020 4:05:00 PM CEST       Success
Patch location validation                July 26, 2020 4:05:00 PM CEST       July 26, 2020 4:05:00 PM CEST       Success
dcs-cli upgrade                          July 26, 2020 4:05:02 PM CEST       July 26, 2020 4:05:02 PM CEST       Success
dcs-cli upgrade                          July 26, 2020 4:05:02 PM CEST       July 26, 2020 4:05:03 PM CEST       Success
Creating repositories using yum          July 26, 2020 4:05:07 PM CEST       July 26, 2020 4:05:10 PM CEST       Success
Updating YumPluginVersionLock rpm        July 26, 2020 4:05:11 PM CEST       July 26, 2020 4:05:11 PM CEST       Success
Applying OS Patches                      July 26, 2020 4:05:11 PM CEST       July 26, 2020 4:24:31 PM CEST       Success
Creating repositories using yum          July 26, 2020 4:24:32 PM CEST       July 26, 2020 4:24:34 PM CEST       Success
Applying HMP Patches                     July 26, 2020 4:24:34 PM CEST       July 26, 2020 4:24:35 PM CEST       Success
Patch location validation                July 26, 2020 4:24:36 PM CEST       July 26, 2020 4:24:36 PM CEST       Success
Patch location validation                July 26, 2020 4:24:36 PM CEST       July 26, 2020 4:24:36 PM CEST       Success
oda-hw-mgmt upgrade                      July 26, 2020 4:24:37 PM CEST       July 26, 2020 4:25:11 PM CEST       Success
oda-hw-mgmt upgrade                      July 26, 2020 4:25:11 PM CEST       July 26, 2020 4:25:47 PM CEST       Success
OSS Patching                             July 26, 2020 4:25:47 PM CEST       July 26, 2020 4:25:49 PM CEST       Success
Applying Firmware Disk Patches           July 26, 2020 4:26:57 PM CEST       July 26, 2020 4:27:59 PM CEST       Success
Applying Firmware Expander Patches       July 26, 2020 4:28:13 PM CEST       July 26, 2020 4:28:30 PM CEST       Success
Applying Firmware Controller Patches     July 26, 2020 4:28:44 PM CEST       July 26, 2020 4:28:59 PM CEST       Success
Checking Ilom patch Version              July 26, 2020 4:29:01 PM CEST       July 26, 2020 4:29:03 PM CEST       Success
Checking Ilom patch Version              July 26, 2020 4:29:03 PM CEST       July 26, 2020 4:29:06 PM CEST       Success
Patch location validation                July 26, 2020 4:29:06 PM CEST       July 26, 2020 4:29:07 PM CEST       Success
Patch location validation                July 26, 2020 4:29:06 PM CEST       July 26, 2020 4:29:07 PM CEST       Success
Save password in Wallet                  July 26, 2020 4:29:09 PM CEST       July 26, 2020 4:29:09 PM CEST       Success
Apply Ilom patch                         July 26, 2020 4:29:09 PM CEST       July 26, 2020 4:29:11 PM CEST       Success
Apply Ilom patch                         July 26, 2020 4:29:11 PM CEST       July 26, 2020 4:29:12 PM CEST       Success
Copying Flash Bios to Temp location      July 26, 2020 4:29:12 PM CEST       July 26, 2020 4:29:12 PM CEST       Success
Copying Flash Bios to Temp location      July 26, 2020 4:29:12 PM CEST       July 26, 2020 4:29:12 PM CEST       Success
Starting the clusterware                 July 26, 2020 4:31:30 PM CEST       July 26, 2020 4:33:32 PM CEST       Success
clusterware patch verification           July 26, 2020 4:33:38 PM CEST       July 26, 2020 4:33:45 PM CEST       Success
clusterware patch verification           July 26, 2020 4:33:38 PM CEST       July 26, 2020 4:33:45 PM CEST       Success
Patch location validation                July 26, 2020 4:33:45 PM CEST       July 26, 2020 4:33:53 PM CEST       Success
Patch location validation                July 26, 2020 4:33:45 PM CEST       July 26, 2020 4:33:53 PM CEST       Success
Opatch update                            July 26, 2020 4:34:37 PM CEST       July 26, 2020 4:34:41 PM CEST       Success
Opatch update                            July 26, 2020 4:34:37 PM CEST       July 26, 2020 4:34:40 PM CEST       Success
Patch conflict check                     July 26, 2020 4:34:42 PM CEST       July 26, 2020 4:36:18 PM CEST       Success
Patch conflict check                     July 26, 2020 4:36:18 PM CEST       July 26, 2020 4:37:54 PM CEST       Success
clusterware upgrade                      July 26, 2020 4:38:09 PM CEST       July 26, 2020 5:26:26 PM CEST       Success
clusterware upgrade                      July 26, 2020 5:26:27 PM CEST       July 26, 2020 6:14:40 PM CEST       Success
Updating GiHome version                  July 26, 2020 6:14:40 PM CEST       July 26, 2020 6:15:03 PM CEST       Success
Updating GiHome version                  July 26, 2020 6:14:40 PM CEST       July 26, 2020 6:15:03 PM CEST       Success
Update System version                    July 26, 2020 6:16:31 PM CEST       July 26, 2020 6:16:32 PM CEST       Success
Update System version                    July 26, 2020 6:16:32 PM CEST       July 26, 2020 6:16:32 PM CEST       Success
preRebootNode Actions                    July 26, 2020 6:16:32 PM CEST       July 26, 2020 6:17:15 PM CEST       Success
preRebootNode Actions                    July 26, 2020 6:17:15 PM CEST       July 26, 2020 6:18:00 PM CEST       Success
Reboot Ilom                              July 26, 2020 6:18:01 PM CEST       July 26, 2020 6:18:01 PM CEST       Success
Reboot Ilom                              July 26, 2020 6:18:01 PM CEST       July 26, 2020 6:18:02 PM CEST       Success

[root@odat1 ~]#

In ODAX5-2 this process took around two hours. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like and steps executed. Some example:

...
2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.a.u.AgentCommonUtils: dbGiHome version:19.0.0.0:
2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.a.r.s.p.PatchingOperations: Is 18c Database:is18c
2020-07-26 16:54:07,687 DEBUG [clusterware upgrade : JobId=91abf816-b171-4e2f-a206-23013f53fb8e] [] c.o.d.c.u.CommonsUtils:
run: cmd= '[/u01/app/19.0.0.0/grid/OPatch/opatchauto,
 apply,
 /u01/patching/19.7.0.0.200414/31145668,
 -oh,
 /u01/app/19.0.0.0/grid,
 -ocmrf,
 /u01/app/19.0.0.0/grid/default-ocm.rsp]'
2020-07-26 16:54:07,900 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2020-07-26 16:54:07,900 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output :
OPatchauto session is initiated at Sun Jul 26 16:54:07 2020
2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output :
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-07-26_04-54-12PM.log.
2020-07-26 16:54:13,077 DEBUG [Thread-1116] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
...

After you finish this jobIB, and around 10 minutes later, ODA nodes will reboot to boot the new version.

Update-storage

The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not update all the firmawares.

You can see from odacli describe-component that some part are not up to date:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.7.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
19.7.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.7.0.0.0            up-to-date

GI                                        19.7.0.0.200414       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200414
}

DCSAGENT                                  19.7.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           16.00.08.00
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date

System node Name
---------------
odat2

Local System Version
---------------
19.7.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.7.0.0.0            up-to-date

GI                                        19.7.0.0.200414       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200414
}

DCSAGENT                                  19.7.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           16.00.08.00
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date


[root@odat1 ~]#

Look above that diskcontrolers are not with the last version of the firmware. So, we need to call odacli update-storage to patch everything:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.7.0.0.0
{
  "jobId" : "095e10f4-cefc-410d-af3f-b3c8f9cd0af9",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "July 26, 2020 18:35:00 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "July 26, 2020 18:35:00 PM CEST"
}
[root@odat1 ~]#

And we can follow the jobID:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "095e10f4-cefc-410d-af3f-b3c8f9cd0af9"

Job details
----------------------------------------------------------------
                     ID:  095e10f4-cefc-410d-af3f-b3c8f9cd0af9
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  July 26, 2020 6:35:00 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           July 26, 2020 6:39:37 PM CEST       July 26, 2020 6:42:40 PM CEST       Success
Applying Firmware Controller Patches     July 26, 2020 6:49:16 PM CEST       July 26, 2020 6:55:47 PM CEST       Success
preRebootNode Actions                    July 26, 2020 6:55:48 PM CEST       July 26, 2020 6:55:48 PM CEST       Success
preRebootNode Actions                    July 26, 2020 6:55:48 PM CEST       July 26, 2020 6:55:48 PM CEST       Success
Reboot Ilom                              July 26, 2020 6:55:48 PM CEST       July 26, 2020 6:55:48 PM CEST       Success
Reboot Ilom                              July 26, 2020 6:55:48 PM CEST       July 26, 2020 6:55:49 PM CEST       Success

[root@odat1 ~]#

After it finishes, the ODA nodes will reboot again. To reboot be effective maybe you need to wait 10 minutes or more.

And after the reboot you can see that everything is up-to-date:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.7.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
19.7.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.7.0.0.0            up-to-date

GI                                        19.7.0.0.200414       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200414
}

DCSAGENT                                  19.7.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 16.00.08.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date

System node Name
---------------
odat2

Local System Version
---------------
19.7.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.7.0.0.0            up-to-date

GI                                        19.7.0.0.200414       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.10.0.0.200414
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.200414
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.200414
}

DCSAGENT                                  19.7.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        7.8                   up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 16.00.08.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

HMP                                       2.4.5.0.1             up-to-date


[root@odat1 ~]#

After that, you can start all databases that you stopped before and continue to use ODA.

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

1 thought on “Patch ODA from 18.3 to 19.8. Part 3 – 19.6 to 19.7

  1. Pingback: Patch ODA from 18.3 to 19.8. Part 4 – 19.7 to 19.8 | Fernando Simon

Leave a Reply

Your email address will not be published. Required fields are marked *