Patch ODA from 18.3 to 19.8. Part 1 – 18.3 to 18.8

The process of patch ODA is not complicated but requires attention over some steps. The 19.6 version was the first that was possible to patch from 18.8 version, and the version that allows upgrades to newer. If you want to go directly to 19.5 you need to reimage of the appliance. In this post, I will cover the process that I made recently to patch from 18.3 to 19.8 version.

The first part covers the upgrade from 18.3 to 18.8.

Patch Matrix

The matrix of what can be done can be found at this post from ODA blog, and you can check below:

Another important detail is to check the MOS note ODA: Quick Reference Matrix for Linux Release and Kernel by ODA Hardware Type and Version (Doc ID 2680219.1) and verify if your hardware is still compatible.

Remember that in this process, the ODA will reboot several times, so, you need to inform your teams that databases will be unavailable during the process.

Actual environment

The environment is ODA X5-2, running the version 18.3. This same environment I reimaged last year, and you can see the steps that I made in a previous post. And it running bare metal (BM).

The first thing to do is to check the actual hardware about errors. If you have, please fix it before the start. Another important detail is to verify if your disks have smartctl warnings. To do that, execute smartctl -q errorsonly -H -l xerror /dev/DISK; If you have “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]” report (or similar warning), please open an SR to replace the disks. Even the warning can lead to an error during the patch.

I recommend you to save outside of ODA some config files from BOTH nodes (to have quick access in case you need):

[root@odat1 ~]# mkdir /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/hosts /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/fstab /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/resolv.conf /nfs/ODA_PATCH_18/bkp_files_odat1
[root@odat1 ~]# cp /etc/sysconfig/network-scripts/* /nfs/ODA_PATCH_18/bkp_files_odat1/
cp: omitting directory `/etc/sysconfig/network-scripts/backupifcfgFiles'
cp: omitting directory `/etc/sysconfig/network-scripts/bkupIfcfgOrig'
[root@odat1 ~]#
[root@odat1 ~]# cp /u01/app/18.0.0.0/grid/network/admin/listener* /nfs/ODA_PATCH_18/bkp_files_odat1/
[root@odat1 ~]#

Another detail before the start is to check the actual components and running versions. To do that we execute the odacli describe-component:

[root@odat1 ~]# odacli describe-component
System Version
---------------
18.3.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       up-to-date
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       up-to-date
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       up-to-date
}

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      4.0.2.26.b.r125868    up-to-date

BIOS                                      30130500              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d15,c2d0,c2d15 ]                 PD51                  pag1
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7,     PAG1                  up-to-date
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

System node Name
---------------
odat2

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       up-to-date
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       up-to-date
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       up-to-date
}

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      4.0.2.26.b.r125868    up-to-date

BIOS                                      30130500              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d15,c2d0,c2d15 ]                 PD51                  pag1
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7,     PAG1                  up-to-date
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}


[root@odat1 ~]#

Check above that some disks are running the firmware PD51. This occurs because they were changed after the 18.3 version and the firmware is newer than the one that exists at 18.3. If you have something that you want to update you can call the odacli update-storage to do the update. Remember that doing this the ODA nodes can reboot.

18.8

The first thing to do is download the patch ODA of 18.8 version using the patch 30518425. since you will have some reboot during the process, is required to stop the databases. I recommend stopping all databases running to speed up the process. This reduces the load over ASM and to stop/start GI (done internally during the process).

Another point is about HAIP. The HAIP needs to be disabled after you patch the 18.8. If you are coming from a reimaged 18.3 version (like me in this case), the patch process will disable HAIP automatically (since it was never configured), but if your 18.3 came because you already upgrade is from 12.x version, you need to manually disable it.

The process to disable it is not quite simple because evolve CRS and cluster definitions. I made this for Exadata/ZDLRA and you can check in this previous post. To ODA you can check this post. Or read the ODA documentation. Disable HAIP is done after patch to 18.8, and not before.

Update Repository

The first step is to upload the unzipped patch files at the internal ODA repository. This process is done using the odacli update-repository and you pass the full path to unzip patch files (here my files are in NFS folder):

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-repository -f /nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip
{
  "jobId" : "3368e5ba-f5da-4e07-af69-f9f51c29517e",
  "status" : "Created",
  "message" : "/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip",
  "reports" : [ ],
  "createTimestamp" : "July 25, 2020 10:10:35 AM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "July 25, 2020 10:10:35 AM CEST"
}
[root@odat1 18.8]#

This generates one jobId that you can follow until it ends using odacli describe-job:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i  "3368e5ba-f5da-4e07-af69-f9f51c29517e"

Job details
----------------------------------------------------------------
                     ID:  3368e5ba-f5da-4e07-af69-f9f51c29517e
            Description:  Repository Update
                 Status:  Success
                Created:  July 25, 2020 10:10:35 AM CEST
                Message:  /nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server1of2.zip,/nfs/ODA_PATCH_18/18.8/oda-sm-18.8.0.0.0-200209-server2of2.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     July 25, 2020 10:10:36 AM CEST      July 25, 2020 10:10:36 AM CEST      Success
Setting up ssh equivalance               July 25, 2020 10:10:37 AM CEST      July 25, 2020 10:10:37 AM CEST      Success
Copy BundleFile                          July 25, 2020 10:10:38 AM CEST      July 25, 2020 10:13:41 AM CEST      Success
Validating CopiedFile                    July 25, 2020 10:13:41 AM CEST      July 25, 2020 10:14:05 AM CEST      Success
Unzip bundle                             July 25, 2020 10:14:05 AM CEST      July 25, 2020 10:29:48 AM CEST      Success
Unzip bundle                             July 25, 2020 10:29:48 AM CEST      July 25, 2020 10:31:43 AM CEST      Success
Delete PatchBundles                      July 25, 2020 10:31:43 AM CEST      July 25, 2020 10:31:44 AM CEST      Success
Removing ssh keys                        July 25, 2020 10:31:44 AM CEST      July 25, 2020 10:53:37 AM CEST      Success

[root@odat1 18.8]#

DCSAgent

The next step is to update the internal DCS agent for ODA 18.3. This is a simple step done using the odacli update-dcsagent. Check that the version 18.8.0.0 was specified as a parameter:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 18.8.0.0.0
{
  "jobId" : "38eed874-b2ab-4124-934a-e81c26376587",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "July 25, 2020 10:58:14 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "July 25, 2020 10:58:14 AM CEST"
}
[root@odat1 18.8]#

And again, you can check the jobId and wait it finish:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587"

Job details
----------------------------------------------------------------
                     ID:  38eed874-b2ab-4124-934a-e81c26376587
            Description:  DcsAgent patching
                 Status:  Success
                Created:  July 25, 2020 10:58:14 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade                        July 25, 2020 10:58:15 AM CEST      July 25, 2020 11:00:01 AM CEST      Success
dcs-agent upgrade                        July 25, 2020 11:00:01 AM CEST      July 25, 2020 11:01:35 AM CEST      Success
Update System version                    July 25, 2020 11:01:37 AM CEST      July 25, 2020 11:01:37 AM CEST      Success
Update System version                    July 25, 2020 11:01:37 AM CEST      July 25, 2020 11:01:37 AM CEST      Success

[root@odat1 18.8]#

Don’t worry because this update will restart the DCS and for a moment the odacli will not work and can report errors (again, don’t worry, it returns after a little time):

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587"
DCS-10001:Internal error encountered: Fail to get credential .
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "38eed874-b2ab-4124-934a-e81c26376587"
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
[root@odat1 18.8]#

DCSAdmin

The next step will be to update the dcsadmin using the command oracle update-dcsadmin:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 18.8.0.0.0
{
  "jobId" : "78ee826d-4266-4983-a7d8-269ccc959620",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "July 25, 2020 11:03:34 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "July 25, 2020 11:03:34 AM CEST"
}
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "78ee826d-4266-4983-a7d8-269ccc959620"

Job details
----------------------------------------------------------------
                     ID:  78ee826d-4266-4983-a7d8-269ccc959620
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  July 25, 2020 11:03:34 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 25, 2020 11:03:36 AM CEST      July 25, 2020 11:03:36 AM CEST      Success
Patch location validation                July 25, 2020 11:03:36 AM CEST      July 25, 2020 11:03:36 AM CEST      Success
dcs-admin upgrade                        July 25, 2020 11:03:37 AM CEST      July 25, 2020 11:03:38 AM CEST      Success
dcs-admin upgrade                        July 25, 2020 11:03:38 AM CEST      July 25, 2020 11:03:39 AM CEST      Success
Update System version                    July 25, 2020 11:03:40 AM CEST      July 25, 2020 11:03:41 AM CEST      Success
Update System version                    July 25, 2020 11:03:41 AM CEST      July 25, 2020 11:03:41 AM CEST      Success

[root@odat1 18.8]#

DCSComponents

The next is to update the dcscomponents with command odacli update-dcscomponents. And here one detail. The commands will generate one jobID, but will be invalid and you need to use the command “odacli list-jobs” to see the jobs (will be just updated SSH keys – two jobs):

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 18.8.0.0.0
{
  "jobId" : "513d54f5-2f84-433d-8b60-8cdd705a29c5",
  "status" : "Success",
  "message" : null,
  "reports" : null,
  "createTimestamp" : "July 25, 2020 11:05:56 AM CEST",
  "description" : "Job completed and is not part of Agent job list",
  "updatedTime" : "July 25, 2020 11:05:56 AM CEST"
}
[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i 513d54f5-2f84-433d-8b60-8cdd705a29c5
DCS-10000:Resource Job with ID 513d54f5-2f84-433d-8b60-8cdd705a29c5 is not found.
[root@odat1 18.8]#
[root@odat1 18.8]# odacli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
...
...
3368e5ba-f5da-4e07-af69-f9f51c29517e     Repository Update                                                           July 25, 2020 10:10:35 AM CEST      Success
38eed874-b2ab-4124-934a-e81c26376587     DcsAgent patching                                                           July 25, 2020 10:58:14 AM CEST      Success
78ee826d-4266-4983-a7d8-269ccc959620     DcsAdmin patching                                                           July 25, 2020 11:03:34 AM CEST      Success
bcaff693-05f5-4a47-b919-a7c40b4f5616     SSH keys update                                                             July 25, 2020 11:06:19 AM CEST      Success
07e98fb0-a334-483f-a772-f838f48228bf     SSH key delete                                                              July 25, 2020 11:06:24 AM CEST      Success

[root@odat1 18.8]#

Create-prepatchreport

The next step is to verify if you can update it to 18.8 or no. The odacli will do a lot of tests for the operational system, GI, and ILOM. To do that, we just need to call odacli create-prepatchreport, and after that use the odacli describe-prepatchreport to check the report.

Some known issues can be checked before. One that I passed was the existence of /u01/app/oraInventory/locks. I needed to remove it from both nodes to have success at the report. Besides I receive:

__GI__
Validate supported GI versions  Success   Validated minimum supported versions
Validate available space        Success   Validated free space under /u01
Verify DB Home versions         Success   Verified DB Home versions
Validate patching locks         Failed    Internal error encountered:
                                          /u01/app/oraInventory/locks.



[root@odat1 18.8]#

The call for patchreport:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 18.8.0.0.0

Job details
----------------------------------------------------------------
                     ID:  3d7c2782-a044-4db2-a912-4e9587920438
            Description:  Patch pre-checks for [OS, ILOM, GI]
                 Status:  Created
                Created:  July 25, 2020 11:25:39 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odat1 18.8]#
[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-prepatchreport -i 3d7c2782-a044-4db2-a912-4e9587920438

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  3d7c2782-a044-4db2-a912-4e9587920438
            Description:  Patch pre-checks for [OS, ILOM, GI]
                 Status:  SUCCESS
                Created:  July 25, 2020 11:25:39 AM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
odat1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions
Validate patching tag           Success   Validated patching tag: 18.8.0.0.0
Is patch location available     Success   Patch location is available
Verify OS patch                 Success   There are no packages available for
                                          an update

__ILOM__
Validate supported versions     Success   Validated minimum supported versions
Validate patching tag           Success   Validated patching tag: 18.8.0.0.0
Is patch location available     Success   Patch location is available
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location

__GI__
Validate supported GI versions  Success   Validated minimum supported versions
Validate available space        Success   Validated free space under /u01
Verify DB Home versions         Success   Verified DB Home versions
Validate patching locks         Success   Validated patching locks

Node Name
---------------
odat2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions
Validate patching tag           Success   Validated patching tag: 18.8.0.0.0
Is patch location available     Success   Patch location is available
Verify OS patch                 Success   There are no packages available for
                                          an update

__ILOM__
Validate supported versions     Success   Validated minimum supported versions
Validate patching tag           Success   Validated patching tag: 18.8.0.0.0
Is patch location available     Success   Patch location is available
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location

__GI__
Validate supported GI versions  Success   Validated minimum supported versions
Validate available space        Success   Validated free space under /u01
Verify DB Home versions         Success   Verified DB Home versions
Validate patching locks         Success   Validated patching locks



[root@odat1 18.8]#

With success results, we can continue the patch. Without that, don’t try to continue (besides if there is a known issue that can be ignored – we will talk about that later).

Stop TFA

Before continue is a requirement stop TFA in both nodes. This is needed because will be changed the version and upgraded to AHF. So, we call /etc/init.d/init.tfa stop in both nodes:

[root@odat1 18.8]# /etc/init.d/init.tfa stop
Stopping TFA from init for shutdown/reboot
oracle-tfa stop/waiting
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Nothing to do !
Killing TFA running with pid 20464
. . .
Successfully stopped TFA..
[root@odat1 18.8]#

###############################################################
#NODE 2
###############################################################
[root@odat2 ~]# /etc/init.d/init.tfa stop
Stopping TFA from init for shutdown/reboot
oracle-tfa stop/waiting
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Nothing to do !
Killing TFA running with pid 49871
. . .
Successfully stopped TFA..
[root@odat2 ~]#

Update-server

After the success of the report, we can call the odacli update-server. And here the magic occurs, the O.S. is updated, GI, and ILOM too:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli update-server -v 18.8.0.0.0
{
  "jobId" : "5d27346e-1afd-4a1c-9e79-3e3b5d523244",
  "status" : "Created",
  "message" : "Success of Server Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "July 25, 2020 11:31:56 AM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "July 25, 2020 11:31:56 AM CEST"
}
[root@odat1 18.8]#

As you imagine, a jobIB is generated and we can follow it until it finishes:

[root@odat1 18.8]# /opt/oracle/dcs/bin/odacli describe-job -i "5d27346e-1afd-4a1c-9e79-3e3b5d523244"

Job details
----------------------------------------------------------------
                     ID:  5d27346e-1afd-4a1c-9e79-3e3b5d523244
            Description:  Server Patching
                 Status:  Success
                Created:  July 25, 2020 11:31:56 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 25, 2020 11:32:08 AM CEST      July 25, 2020 11:32:08 AM CEST      Success
Patch location validation                July 25, 2020 11:32:08 AM CEST      July 25, 2020 11:32:08 AM CEST      Success
dcs-controller upgrade                   July 25, 2020 11:32:08 AM CEST      July 25, 2020 11:32:13 AM CEST      Success
dcs-controller upgrade                   July 25, 2020 11:32:13 AM CEST      July 25, 2020 11:32:18 AM CEST      Success
Patch location validation                July 25, 2020 11:32:21 AM CEST      July 25, 2020 11:32:21 AM CEST      Success
Patch location validation                July 25, 2020 11:32:21 AM CEST      July 25, 2020 11:32:21 AM CEST      Success
dcs-cli upgrade                          July 25, 2020 11:32:22 AM CEST      July 25, 2020 11:32:23 AM CEST      Success
dcs-cli upgrade                          July 25, 2020 11:32:23 AM CEST      July 25, 2020 11:32:24 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:33 AM CEST      July 25, 2020 11:32:41 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:41 AM CEST      July 25, 2020 11:32:41 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:42 AM CEST      July 25, 2020 11:32:42 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:42 AM CEST      July 25, 2020 11:32:42 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:43 AM CEST      July 25, 2020 11:32:43 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:43 AM CEST      July 25, 2020 11:32:43 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:32:43 AM CEST      July 25, 2020 11:32:43 AM CEST      Success
Updating YumPluginVersionLock rpm        July 25, 2020 11:32:44 AM CEST      July 25, 2020 11:32:44 AM CEST      Success
Applying OS Patches                      July 25, 2020 11:32:45 AM CEST      July 25, 2020 11:45:19 AM CEST      Success
Creating repositories using yum          July 25, 2020 11:45:20 AM CEST      July 25, 2020 11:45:21 AM CEST      Success
Applying HMP Patches                     July 25, 2020 11:45:21 AM CEST      July 25, 2020 11:45:52 AM CEST      Success
Patch location validation                July 25, 2020 11:45:53 AM CEST      July 25, 2020 11:45:53 AM CEST      Success
Patch location validation                July 25, 2020 11:45:53 AM CEST      July 25, 2020 11:45:53 AM CEST      Success
oda-hw-mgmt upgrade                      July 25, 2020 11:45:54 AM CEST      July 25, 2020 11:46:25 AM CEST      Success
oda-hw-mgmt upgrade                      July 25, 2020 11:46:25 AM CEST      July 25, 2020 11:46:59 AM CEST      Success
OSS Patching                             July 25, 2020 11:47:00 AM CEST      July 25, 2020 11:47:00 AM CEST      Success
Applying Firmware Disk Patches           July 25, 2020 11:47:27 AM CEST      July 25, 2020 11:47:42 AM CEST      Success
Applying Firmware Expander Patches       July 25, 2020 11:48:01 AM CEST      July 25, 2020 11:48:13 AM CEST      Success
Applying Firmware Controller Patches     July 25, 2020 11:48:31 AM CEST      July 25, 2020 11:48:41 AM CEST      Success
Checking Ilom patch Version              July 25, 2020 11:48:43 AM CEST      July 25, 2020 11:48:46 AM CEST      Success
Checking Ilom patch Version              July 25, 2020 11:48:46 AM CEST      July 25, 2020 11:48:48 AM CEST      Success
Patch location validation                July 25, 2020 11:48:48 AM CEST      July 25, 2020 11:48:49 AM CEST      Success
Patch location validation                July 25, 2020 11:48:48 AM CEST      July 25, 2020 11:48:49 AM CEST      Success
Save password in Wallet                  July 25, 2020 11:48:51 AM CEST      July 25, 2020 11:48:51 AM CEST      Success
Apply Ilom patch                         July 25, 2020 11:48:52 AM CEST      July 25, 2020 12:01:47 PM CEST      Success
Apply Ilom patch                         July 25, 2020 12:01:47 PM CEST      July 25, 2020 12:14:11 PM CEST      Success
Copying Flash Bios to Temp location      July 25, 2020 12:14:11 PM CEST      July 25, 2020 12:14:12 PM CEST      Success
Copying Flash Bios to Temp location      July 25, 2020 12:14:12 PM CEST      July 25, 2020 12:14:12 PM CEST      Success
Starting the clusterware                 July 25, 2020 12:15:57 PM CEST      July 25, 2020 12:17:24 PM CEST      Success
clusterware patch verification           July 25, 2020 12:24:48 PM CEST      July 25, 2020 12:24:50 PM CEST      Success
clusterware patch verification           July 25, 2020 12:24:48 PM CEST      July 25, 2020 12:24:50 PM CEST      Success
Patch location validation                July 25, 2020 12:24:50 PM CEST      July 25, 2020 12:25:03 PM CEST      Success
Patch location validation                July 25, 2020 12:24:50 PM CEST      July 25, 2020 12:25:03 PM CEST      Success
Opatch updation                          July 25, 2020 12:25:48 PM CEST      July 25, 2020 12:25:53 PM CEST      Success
Opatch updation                          July 25, 2020 12:25:48 PM CEST      July 25, 2020 12:25:54 PM CEST      Success
Patch conflict check                     July 25, 2020 12:25:54 PM CEST      July 25, 2020 12:27:04 PM CEST      Success
Patch conflict check                     July 25, 2020 12:27:07 PM CEST      July 25, 2020 12:28:23 PM CEST      Success
clusterware upgrade                      July 25, 2020 12:28:24 PM CEST      July 25, 2020 12:50:42 PM CEST      Success
clusterware upgrade                      July 25, 2020 12:50:42 PM CEST      July 25, 2020 1:15:15 PM CEST       Success
Updating GiHome version                  July 25, 2020 1:15:16 PM CEST       July 25, 2020 1:15:21 PM CEST       Success
Updating GiHome version                  July 25, 2020 1:15:16 PM CEST       July 25, 2020 1:15:21 PM CEST       Success
Update System version                    July 25, 2020 1:16:21 PM CEST       July 25, 2020 1:16:21 PM CEST       Success
Update System version                    July 25, 2020 1:16:22 PM CEST       July 25, 2020 1:16:22 PM CEST       Success
preRebootNode Actions                    July 25, 2020 1:16:22 PM CEST       July 25, 2020 1:17:04 PM CEST       Success
preRebootNode Actions                    July 25, 2020 1:17:04 PM CEST       July 25, 2020 1:17:45 PM CEST       Success
Reboot Ilom                              July 25, 2020 1:17:46 PM CEST       July 25, 2020 1:17:46 PM CEST       Success
Reboot Ilom                              July 25, 2020 1:17:46 PM CEST       July 25, 2020 1:17:46 PM CEST       Success

[root@odat1 18.8]#

In ODAX5-2 this process took al most two hours. And passed all the updates. If you follow the /opt/oracle/dcs/log/dcs-agent.log you can see more detail like:

...  
...
2020-07-25 12:20:14,275 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_
2020-07-25 12:20:16,276 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_
2020-07-25 12:20:18,277 INFO [Disable haip feature : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.a.z.DCSZQueue: Node not yet received for: /nodes/node_1/cmd-out-q with prefix:323_5973_
...
...
2020-07-25 12:25:54,991 DEBUG [Patch conflict check : JobId=5d27346e-1afd-4a1c-9e79-3e3b5d523244] [] c.o.d.c.u.CommonsUtils:
run: cmd= '[/u01/app/18.0.0.0/grid/OPatch/opatchauto,
 apply,
 /u01/patching/18.8.0.0.191015/30518456,
 -analyze,
 -oh,
 /u01/app/18.0.0.0/grid,
 -log,
 /tmp/opatchAutoAnalyzePatch.log,
 -inplace]'
...
...

After this jobID finishes, the ODA will reboot (between the end of the job and reboot can be around 10 minutes).

Update-storage

The last step is to patch the storage with command odacli update-storage. This is needed because the previous step does not updates the firmware of the disks as an example.

You can see from odacli describe-component that some part are not up to date:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
18.8.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
18.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.8.0.0.0            up-to-date

GI                                        18.8.0.0.191015       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.191015
}

DCSAGENT                                  18.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  001e

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d15,c2d0,c2d15 ]                 PD51                  up-to-date
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7,     PAG1                  pd51
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

System node Name
---------------
odat2

Local System Version
---------------
18.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.8.0.0.0            up-to-date

GI                                        18.8.0.0.191015       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.191015
}

DCSAGENT                                  18.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  001e

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d15,c2d0,c2d15 ]                 PD51                  up-to-date
[ c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,c1d7,     PAG1                  pd51
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,
c2d9,c2d10,c2d11,c2d12,c2d13,c2d14 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}


[root@odat1 ~]#

And we can update calling the odacli update-storage:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli update-storage -v 18.8.0.0.0
{
  "jobId" : "06bbdc8d-d7d2-4400-82cc-a3345ceca28d",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "July 25, 2020 14:35:15 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "July 25, 2020 14:35:15 PM CEST"
}
[root@odat1 ~]#
[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i "06bbdc8d-d7d2-4400-82cc-a3345ceca28d"

Job details
----------------------------------------------------------------
                     ID:  06bbdc8d-d7d2-4400-82cc-a3345ceca28d
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  July 25, 2020 2:35:15 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           July 25, 2020 2:49:02 PM CEST       July 25, 2020 2:49:25 PM CEST       Success
Applying Firmware Expander Patches       July 25, 2020 2:53:03 PM CEST       July 25, 2020 2:56:38 PM CEST       Success
Applying Firmware Controller Patches     July 25, 2020 2:56:51 PM CEST       July 25, 2020 2:57:08 PM CEST       Success
preRebootNode Actions                    July 25, 2020 2:57:09 PM CEST       July 25, 2020 2:57:09 PM CEST       Success
preRebootNode Actions                    July 25, 2020 2:57:09 PM CEST       July 25, 2020 2:57:09 PM CEST       Success
Reboot Ilom                              July 25, 2020 2:57:09 PM CEST       July 25, 2020 2:57:09 PM CEST       Success
Reboot Ilom                              July 25, 2020 2:57:10 PM CEST       July 25, 2020 2:57:10 PM CEST       Success

[root@odat1 ~]#

After the job finish (and after around 10 minutes) the ODA will reboot automatically.

And you can see that now that everything is up to date:

[root@odat1 ~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
18.8.0.0.0

System node Name
---------------
odat1

Local System Version
---------------
18.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.8.0.0.0            up-to-date

GI                                        18.8.0.0.191015       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.191015
}

DCSAGENT                                  18.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

System node Name
---------------
odat2

Local System Version
---------------
18.8.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.8.0.0.0            up-to-date

GI                                        18.8.0.0.191015       up-to-date

DB {
[ OraDB18000_home1 ]                      18.3.0.0.180717       18.8.0.0.191015
[ OraDB12102_home1,OraDB12102_home2 ]     12.1.0.2.180717       12.1.0.2.191015
[ OraDB11204_home1,OraDB11204_home2 ]     11.2.0.4.180717       11.2.0.4.191015
}

DCSAGENT                                  18.8.0.0.0            up-to-date

ILOM                                      4.0.4.52.r132805      up-to-date

BIOS                                      30300200              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          001E                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PD51                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}


[root@odat1 ~]#

If you check the log, you can see that firmware was updated:

2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.r.s.p.PatchingOperations: Firmware Disk Patch File---->/opt/oracle/oak/pkgrepos/firmwaredisk/hgst/h7280a520sun8.0t/pd51/H7280A520.PD51.fw
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: StorageUtils::isBRCMProduct
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Firmware version to be applied---->PD51
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Storage Component Current Firmware Version---->PAG1
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Known Firmware Versions------>P554,P901,P9E2,PAG1,PD51
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: StorageUtils::isBRCMProduct
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.u.s.StorageUtils: Storage Component Patch Applicable---->c2d3
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.a.r.s.g.GiUtils: GiUtils:isCrsRunning
2020-07-25 14:44:45,386 DEBUG [TaskZJsonRpcExt_256 : JobId=06bbdc8d-d7d2-4400-82cc-a3345ceca28d] [] c.o.d.c.u.CommonsUtils:

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

1 thought on “Patch ODA from 18.3 to 19.8. Part 1 – 18.3 to 18.8

  1. Pingback: Patch ODA from 18.3 to 19.8. Part 2 – 18.8 to 19.6 | Fernando Simon

Leave a Reply

Your email address will not be published. Required fields are marked *