ZDLRA, Patch/Update the Recovery Appliance

The process of patch ZDLRA is not complicated, but it is important to be aware of some details. The most important is from where you are until where you want to go. This is crucial because it will define what commands you will need to execute.

If you read the previous post about the process, you can notice that I was running the ZDLRA 12.2 version, and forwarded to 19.2 version. In that case, I needed to use the upgrade path since I was changing the major release and the racli commands had the “upgrade” parameter.

In this post I will show how to do a simple update (or patch apply) for ZDLRA, this means that I will remain inside the same major release for recovery appliance library. Some steps and checks are the same.

Whatever you need to do (patch or upgrade), the startup point it is the note 1927416.1 that cover the supported versions for ZDLRA. There it is possible to find all the supported versions for the recovery appliance library as well as the Exadata versions. Please, not upgrade the Exadata stack with a version that is not listed on this page.

Where we are

The first step is always to discover what version for ZDLRA and Exadata Stack you are running. The simple way to do that is using the command “racli version”:

[root@zeroinsg01 ~]# racli version
Recovery Appliance Version:
        exadata image: 19.2.3.0.0.190621
        rarpm version: ra_automation-19.2.1.1.1.202001-31014797.x86_64
        rdbms version: RDBMS_19.3.0.0.190416DBRU_LINUX.X64_RELEASE
        transaction  : kadjei_bug-31014797
        zdlra version: ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE
[root@zeroinsg01 ~]#

The way to read this is that: rpm version is the running version, zdlra version is the base release (202001). The major release is 19.2.1.1.1. So, it is possible to discover that the currently running version for ZDLRA is 19.2.1.1.1.202001-31014797 and the Exadata image is 19.2.3.0.0.190621.

Where we want to go

As I wrote before, the startup point it is the note 1927416.1. There we can check the available versions for ZDLRA, and in this example, I will patch to version 19.2.1.1.1-202003-31304538 (released 18-May-2020).

So, I will jump from 19.2.1.1.1.202001-31014797 to 19.2.1.1.1-202003-31304538, but I will remain inside of the same major release of ZDLRA, the 19.2.1.1.1. I will jump from base release (like PSU for OH) from 202001 (released in January) to 202003 (released in March).

Exadata Supported Versions

Again, checking the note 1927416.1 that covers the supported versions for ZDLRA is possible to check the compatibility matrix for Exadata System Software (database and storage servers) and discover if it is needed to upgrade/patch the Exadata stack.

Look above that the current version for Exadata is supported. So, this means that I can do just the patch apply for ZDLRA. If you want to update the Exadata Stack you can check my post on how to do that, but the process is very, very, similar for Exadata Patch apply.

The only detail is that usually, we update the ZDLRA library first, and after the Exadata Stack. This occurs, (because as you can see in the image above), that the current version that I am running (19.2.1.1.1.202001-31014797) is not compatible with the Exadata 19.3.7.0.0. So, first, update ZDLRA, and after Exadata stack. In this example, I will just update the ZDLRA library.

Patch Process

Download

Since we will continue in the same major release of ZDLRA (19.2.1.1.1), this means that the OH and GI will continue at the same release and will not suffer any kind of version upgrade (like 12 to 19). So, we don’t need to download additional patches or binaries. In my previous post, I already explained this.

As you know, the ZDLRA library contains the updates for the OH and GI, so the compatible PSU’s will be applied and they are inside of downloaded patched:

Where put the files

As I wrote in my previous post, all the files for ZDLRA need to be stored at /radump at the database server node 01. It is fixed in the procedure and it is a requirement.

And as best practices, I recommend that before copy the new files, remove all older patches from /radump in both nodes. And this include files that are inside the ZDLRA patch (like ra_init_param_check.pl, load_init_param.sh, load_init_param.pl,  dbmsrsadm.sql, dbmsrsadmpreq.sql, prvtrsadm.sql, ra_preinstall.pl).

01 – Precheck

The first step is to do a simple precheck of the ZDLRA stack. The simple and fast way is executing the “racli status appliance”:

[root@zeroinsg01 ~]# racli status appliance
zeroinsg01 crs        state:  [ONLINE]
zeroinsg01 db         state:  [ONLINE]
zeroinsg01 ra_server  state:  [ONLINE]
zeroinsg02 crs        state:  [ONLINE]
zeroinsg02 ra_server  state:  [ONLINE]
zeroinsg02 db         state:  [ONLINE]
[root@zeroinsg01 ~]#

And the “racli run check –all“:

[root@zeroinsg01 ~]# racli run check --all
Wed Jun  3 10:07:21 2020: Start: racli run check --all
  Created log file zeroinsg01.bnet.luxds.net:/opt/oracle.RecoveryAppliance/log/racli_run_check_20200603.1007.log

Wed Jun  3 10:07:26 2020: CHECK: RA Services                         - PASS
Wed Jun  3 10:07:37 2020: CHECK: Exadata Image Version               - PASS
Wed Jun  3 10:07:38 2020: CHECK: Active Incidents                    - PASS
Wed Jun  3 10:07:46 2020: CHECK: Init Parameters                     - PASS
Wed Jun  3 10:07:47 2020: CHECK: Invalid Objects                     - PASS
Wed Jun  3 10:07:48 2020: CHECK: Export Backup                       - PASS
Wed Jun  3 10:07:48 2020: CHECK: ZDLRA Rasys Wallet                  - PASS
Wed Jun  3 10:07:51 2020: CHECK: Compute Node AlertHistory
Wed Jun  3 10:07:51 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 10:07:51 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 10:07:59 2020: CHECK: Storage Cell AlertHistory
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm04]                   - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm02]                   - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm06]                   - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm05]                   - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm01]                   - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zerocadm03]                   - PASS
Wed Jun  3 10:07:59 2020: CHECK: Oracle User Password Expires
Wed Jun  3 10:07:59 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 10:07:59 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 10:08:00 2020: CHECK: ZDLRA Version
Wed Jun  3 10:08:00 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 10:08:00 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 10:08:00 2020: End: racli run check --all
[root@zeroinsg01 ~]#

With that, every error can be checked, reported, and fixed before continuing the process.

02 – Unzip Library

The second step is, as oracle user, unzip the ZDLRA patch. This is done inside of /radump folder at node 01:

[oracle@zeroinsg01 radump]$ unzip p31304538_192111_Linux-x86-64.zip
Archive:  p31304538_192111_Linux-x86-64.zip
  inflating: set_env.sh
 extracting: p6880880_180000_Linux-x86-64.zip
  inflating: dbmsrsadmpreq.sql
  inflating: dbmsrsadm.sql
  inflating: prvtrsadm.sql
  inflating: create_raoratab.pl
  inflating: create_raoratab.sh
  inflating: get_versions.pl
  inflating: get_versions.sh
  inflating: ra_init_param_check.pl
  inflating: ra_init_param_check.sh
  inflating: ra_precheck.pl
  inflating: ra_precheck.sh
  inflating: ra_preinstall.pl
  inflating: ra_automation-19.2.1.1.1.202003-31304538.x86_64.rpm
  inflating: run_set_env.sh
 extracting: p29908639_193000DBRU_Linux-x86-64.zip
 extracting: p29232533_190000_Linux-x86-64.zip
 extracting: p29726449_190000_Linux-x86-64.zip
 extracting: p29391849_194000DBRU_Linux-x86-64.zip
 extracting: p30177140_194000DBRU_Linux-x86-64.zip
 extracting: p30540969_190000_Linux-x86-64.zip
 extracting: p30312546_190000_Linux-x86-64.zip
 extracting: p30458593_190000_Linux-x86-64.zip
 extracting: p30363621_190000_Linux-x86-64.zip
 extracting: p28538439_190000_Linux-x86-64.zip
 extracting: p30094929_190000_Linux-x86-64.zip
 extracting: p29708769_190000_Linux-x86-64.zip
  inflating: README.txt
[oracle@zeroinsg01 radump]$

As you can see the patch p31304538_192111_Linux-x86-64.zip contains inside the PSU/One-offs (p* files) patches for OH and GI.

03 – ra_preinstall.pl

The third is step is to execute the ra_preinstall.pl, as root user, to start the process. This step removes the old ZDLRA rpm library from the system and installs the new one (in both nodes). Besides that, execute some checks as well.

As explained in my previous post, if you have some open incidents you can clean it (the process is described there), but also is not recommended to restart this ZDLRA after this moment because the ZDLRA binaries were changed (rpm was changed) but the database itself was not upgraded. Of course, that library supposes to be/have interoperability, but if it is possible to avoid the problem, it is desired.

[root@zeroinsg01 ~]# cd /radump/
[root@zeroinsg01 radump]#
[root@zeroinsg01 radump]# which perl
/opt/oracle.RecoveryAppliance/bin/perl
[root@zeroinsg01 radump]#
[root@zeroinsg01 radump]# perl ra_preinstall.pl
Start: Running ra_preinstall.pl on zeroinsg01.
NOTE:
    Current deployed RPM [ra_automation-19.2.1.1.1.202001-31014797.x86_64.rpm] not found!
    If you continue without an old RPM, rollback will not be possible.
    Refer to the README.txt included in this ZDLRA Patch for more details.

    Do you want to continue? (y|n):  y
Note:
    The ra_preinstall.pl manages the ra_automation RPM,
    and provides a --rollback option.
    The RPM is updated during ra preinstall. Rollback is feasible if the old rpm is found.
    You do not need to update RPM separately.

    Refer to the README.txt included in this ZDLRA Patch for more details.

    Do you want to continue (y|n):  y
Deployed RPM:  ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE
Installed RPM: ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE
RPM matches.
Fuse group already exists. Skipping.
Start Update sshd_config
End Update sshd_config
End: Running ra_preinstall.pl on zeroinsg01.
Copying ra_preinstall.pl, create_raoratab.* and new RPM to remote node zeroinsg02. Password may be required to connect.
ra_preinstall.pl                                             100%   42KB  62.5MB/s   00:00
create_raoratab.pl                                           100% 3251    13.7MB/s   00:00
create_raoratab.sh                                           100%  991     5.5MB/s   00:00
run_set_env.sh                                               100% 1000     4.4MB/s   00:00
set_env.sh                                                   100% 2420    13.5MB/s   00:00
ra_automation-19.2.1.1.1.202003-31304538.x86_64.rpm          100% 1903MB 227.2MB/s   00:08
Created log /opt/oracle.RecoveryAppliance/log/create_raoratab.log
Start: Running ra_preinstall.pl on zeroinsg02.
Fuse group already exists. Skipping.
Start Update sshd_config
End Update sshd_config
End: Running ra_preinstall.pl on zeroinsg02.
Start Restart sshd
End Restart sshd
Start: Check Init Parameters
Created log file /opt/oracle.RecoveryAppliance/log/ra_init_param_check.log
All init parameters have been validated.
End: Check Init Parameters
Start: Remove current RPM and install new RPM on zeroinsg01.
End: Remove current RPM and install new RPM on zeroinsg01.
Start: Remove current RPM and install new RPM on zeroinsg02.
End: Remove current RPM and install new RPM on zeroinsg02.
Created log /opt/oracle.RecoveryAppliance/log/raprecheck.log
Start: Run Pre check for Patch
Wed Jun  3 10:10:34 2020: Start: Check ZDLRA Services
Wed Jun  3 10:10:38 2020: End: Check ZDLRA Services
Wed Jun  3 10:10:38 2020: Start: Check ASM rebalance
Wed Jun  3 10:10:39 2020: End: Check ASM rebalance
Wed Jun  3 10:10:39 2020: Start: Check Cluster
Wed Jun  3 10:11:48 2020: End: Check Cluster
Wed Jun  3 10:11:53 2020: Start: Check Open Incidents
Wed Jun  3 10:11:54 2020: End: Check Open Incidents
Wed Jun  3 10:11:54 2020: Start: Check Invalid Objects
Wed Jun  3 10:11:58 2020: End: Check Invalid Objects
Wed Jun  3 10:11:58 2020: Start: Check Init Parameters
Wed Jun  3 10:11:59 2020: End: Check Init Parameters
Wed Jun  3 10:11:59 2020: Start: Check compute node oracle access
Wed Jun  3 10:11:59 2020: End: Check compute node oracle access
Wed Jun  3 10:11:59 2020: Start: Check TFA/AHF status
Wed Jun  3 10:12:03 2020: End: Check TFA/AHF status
End: Run Pre check for Patch
Start Restart sshd


!!!NOTE!!!

Exit and log back into [zeroinsg01] prior to continuing


!!!NOTE!!!

End Restart sshd
[root@zeroinsg01 radump]#





Last login: Wed Jun  3 10:12:30 2020 from xw11lapp0041.bnet.luxds.net
[root@zeroinsg01 ~]#

I always check where Perl is coming, be sure that it is from the ZDLRA folder. The warning about rpm file not found means that the current rpm file was not found at /radump (probably cleaned in some moment), but since I have it in local NFS, I can just copy it if needed.

04 – patch appliance –step=1

After the rpm is changed in both nodes, we can call the racli to do continue with the patch. The first step is just a precheck phase. This is executed as root:

[root@zeroinsg01 ~]# /opt/oracle.RecoveryAppliance/bin/racli patch appliance --step=1
Created log /opt/oracle.RecoveryAppliance/log/racli_patch_appliance.log
Wed Jun  3 10:13:36 2020: Start: Patch Recovery Appliance - Step [PreCheck]
Wed Jun  3 10:13:36 2020: Start: Check ZDLRA Services
Wed Jun  3 10:13:40 2020: End: Check ZDLRA Services
Wed Jun  3 10:13:40 2020: Start: Check ASM rebalance
Wed Jun  3 10:13:41 2020: End: Check ASM rebalance
Wed Jun  3 10:13:41 2020: Start: Check Cluster
Wed Jun  3 10:14:57 2020: End: Check Cluster
Wed Jun  3 10:15:00 2020: Start: Check Open Incidents
Wed Jun  3 10:15:01 2020: End: Check Open Incidents
Wed Jun  3 10:15:01 2020: Start: Check Invalid Objects
Wed Jun  3 10:15:05 2020: End: Check Invalid Objects
Wed Jun  3 10:15:05 2020: Start: Check Init Parameters
Wed Jun  3 10:15:07 2020: End: Check Init Parameters
Wed Jun  3 10:15:07 2020: Start: Check compute node oracle access
Wed Jun  3 10:15:07 2020: End: Check compute node oracle access
Wed Jun  3 10:15:07 2020: Start: Check TFA/AHF status
Wed Jun  3 10:15:10 2020: End: Check TFA/AHF status
Wed Jun  3 10:15:13 2020: Start: Check RA Patch Levels
Wed Jun  3 10:15:13 2020: End: Check RA Patch Levels
Wed Jun  3 10:15:13 2020: Start: Check Patches
Wed Jun  3 10:15:25 2020: End: Check Patches
Wed Jun  3 10:15:25 2020: End: Patch Recovery Appliance - Step [PreCheck]
[root@zeroinsg01 ~]#

This check if some errors exist inside ASM or ZDLRA database metadata.

As you can above, the command racli now uses the parameter “patch appliance”. In my previous posts, I used “upgrade appliance”. The patch, as explained, is used when we are inside of the same major release.

05 – patch appliance –step=2

The next step is the patch itself, the racli called as root will apply all patches needed.

[root@zeroinsg01 ~]# /opt/oracle.RecoveryAppliance/bin/racli patch appliance --step=2
Created log /opt/oracle.RecoveryAppliance/log/racli_patch_appliance.log
Wed Jun  3 10:17:46 2020: Start: Patch Recovery Appliance - Step [Patch]
Wed Jun  3 10:17:46 2020: Skip: Patch Recovery Appliance - Step [PreCheck] [Already Run]
Wed Jun  3 10:17:48 2020: Start: Pre Patch Release Steps
Wed Jun  3 10:17:48 2020: End: Pre Patch Release Steps
Wed Jun  3 10:17:48 2020: Start: Unpack Patches
Wed Jun  3 10:19:24 2020: End: Unpack Patches
Wed Jun  3 10:19:24 2020: Start: Rasys configuration
Wed Jun  3 10:19:27 2020: End: Rasys configuration
Wed Jun  3 10:19:27 2020: Start: Stop ZDLRA Services
Wed Jun  3 10:23:26 2020: End: Stop ZDLRA Services
Wed Jun  3 10:23:26 2020: Start: Disable ZDLRA Exports
Wed Jun  3 10:23:26 2020: End: Disable ZDLRA Exports
Wed Jun  3 10:23:26 2020: Start: Check Active Programs
Wed Jun  3 10:23:57 2020: Start: Stop Clusterware
Wed Jun  3 10:24:34 2020: End: Stop Clusterware
Wed Jun  3 10:24:34 2020: Start: Checking active programs
Wed Jun  3 10:24:46 2020: End: Checking active programs
Wed Jun  3 10:24:46 2020: Start: Start Clusterware
Wed Jun  3 10:30:27 2020: End: Start Clusterware
Wed Jun  3 10:30:27 2020: End: Check Active Programs
Wed Jun  3 10:30:27 2020: Start: Patch OPatch
Wed Jun  3 10:30:29 2020: End: Patch OPatch
Wed Jun  3 10:30:29 2020: Start: CRS start
Wed Jun  3 10:32:56 2020: End: CRS start
Wed Jun  3 10:32:56 2020: Start: Installed Patch Check
Wed Jun  3 10:35:40 2020: End: Installed Patch Check
Wed Jun  3 10:35:40 2020: Start: Apply Bundle Patch
Wed Jun  3 10:36:52 2020:       Skipped: Bundle Patch is Current.
Wed Jun  3 10:36:52 2020: End: Apply Bundle Patch
Wed Jun  3 10:36:53 2020: End: Apply Bundle Patch
Wed Jun  3 10:36:53 2020: Start: Apply GI Patches
Wed Jun  3 10:54:56 2020: End: Apply GI Patches
Wed Jun  3 10:54:56 2020: Start: Install ZDLRA Patch
Wed Jun  3 11:00:16 2020: End: Install ZDLRA Patch
Wed Jun  3 11:00:16 2020: Start: Apply Datapatch
Wed Jun  3 11:00:37 2020: End: Apply Datapatch
Wed Jun  3 11:00:37 2020: Start: Rasys Update
Wed Jun  3 11:02:37 2020: End: Rasys Update
Wed Jun  3 11:02:37 2020: Start: ZDLRA OS Updates
Wed Jun  3 11:02:38 2020: End: ZDLRA OS Updates
Wed Jun  3 11:02:38 2020: Start: Exachk Update
Wed Jun  3 11:02:40 2020: End: Exachk Update
Wed Jun  3 11:02:40 2020: Start: AHF Update
Wed Jun  3 11:09:24 2020: End: AHF Update
Wed Jun  3 11:09:28 2020: Start: Secure Backup Update
Wed Jun  3 11:09:28 2020: Start: Tape Update
Wed Jun  3 11:09:58 2020: End: Tape Update
Wed Jun  3 11:09:58 2020: End: Secure Backup Update
Wed Jun  3 11:09:58 2020: Start: Post Patch Service Enable
Wed Jun  3 11:10:01 2020: Start: Stop ZDLRA Stack
Wed Jun  3 11:13:36 2020: End: Stop ZDLRA Stack
Wed Jun  3 11:13:36 2020: Start: Enable ZDLRA
Wed Jun  3 11:13:37 2020: End: Enable ZDLRA
Wed Jun  3 11:18:08 2020: Start: Enable RA Backup
Wed Jun  3 11:18:08 2020: End: Enable RA Backup
Wed Jun  3 11:18:08 2020: End: Post Patch Service Enable
Wed Jun  3 11:18:08 2020: Start: Reset RA user password
Wed Jun  3 11:18:09 2020: End: Reset RA user password
Wed Jun  3 11:18:09 2020: Start: Check for EM Agents
Wed Jun  3 11:19:00 2020: End: Check for EM Agents
Wed Jun  3 11:19:00 2020: Start: Check Init Parameters
Wed Jun  3 11:19:00 2020: End: Check Init Parameters
Wed Jun  3 11:19:00 2020: Start: Setup OS
Wed Jun  3 11:19:35 2020: End: Setup OS
Wed Jun  3 11:19:35 2020: Start: Post Patch Clean Up
Wed Jun  3 11:34:14 2020: End: Post Patch Clean Up
Wed Jun  3 11:34:14 2020: End: Patch Recovery Appliance - Step [Patch]
[root@zeroinsg01 ~]#

The patch will apply everything that is needed: OH Patches, GI Patches, OSB, and ZDLRA stack. In this process, the ZDLRA will be “offline” and not receive ingested backups. I recommend not even try to send backups (or archivelogs) to ZDLRA to avoid concurrencies inside rman catalog (since the patch process touches it too).

You can see below some parts of the patch apply process:

Wed Jun  3 10:17:48 2020: Got /u01/app/oracle/product/19.0.0.0/dbhome_1 patch status
Wed Jun  3 10:17:48 2020: Checking for 29517242.
Wed Jun  3 10:17:48 2020: Found 29517242.
Wed Jun  3 10:17:48 2020: Start: Pre Patch Release Steps
Wed Jun  3 10:17:48 2020: End: RunLevel 1001
Wed Jun  3 10:17:48 2020: Start: RunLevel 1002
Wed Jun  3 10:17:48 2020: End: Pre Patch Release Steps
Wed Jun  3 10:17:48 2020: Start: Unpack Patches
Wed Jun  3 10:17:48 2020: Start: Unpack Patches
Wed Jun  3 10:17:48 2020: Checking for patch [29726449]
Wed Jun  3 10:17:48 2020: Found patch [29726449]. Checking 29726449.
Wed Jun  3 10:17:48 2020: Patch [29726449] md5sum matches. Unpacking 29726449.
Wed Jun  3 10:17:48 2020: Switching to UID: 1001, GID: 1002
Wed Jun  3 10:17:48 2020: Set Command '/usr/bin/unzip -nuq /radump//p29726449_190000_Linux-x86-64.zip -d //radump//patch//general' timeout to 900.

Wed Jun  3 10:17:48 2020: Unpacking 29726449 complete. OK
Wed Jun  3 10:17:48 2020: Checking for patch [29232533]
Wed Jun  3 10:17:48 2020: Found patch [29232533]. Checking 29232533.
Wed Jun  3 10:17:48 2020: Patch [29232533] md5sum matches. Unpacking 29232533.
…
…
Wed Jun  3 10:24:00 2020: spawn /usr/bin/ssh -o ConnectTimeout=20 -o LogLevel=error -l root zeroinsg02 /u01/app/19.0.0.0/grid/bin/crsctl stop crs -f
…
…
Wed Jun  3 10:33:16 2020: Found 28279612.
Wed Jun  3 10:33:16 2020: Removing 28279612
Wed Jun  3 10:33:16 2020: Set Command '/usr/bin/expect ' timeout to 305.

Wed Jun  3 10:33:43 2020: spawn /usr/bin/ssh -o ConnectTimeout=20 -o LogLevel=error -l oracle zeroinsg02 /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 28279612 -local_node -no_relink -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 -silent
Oracle Interim Patch Installer version 12.2.0.1.17
...
...
Wed Jun  3 10:36:52 2020: spawn /usr/bin/ssh -o ConnectTimeout=20 -o LogLevel=error -l root zeroinsg01 /u01/app/19.0.0.0/grid/OPatch/opatchauto apply //radump//patch//gi_opatchauto/29708769 -oh /u01/app/19.0.0.0/grid

OPatchauto session is initiated at Wed Jun  3 10:36:21 2020
…
…
===========================================================
install-zdlra-patch starting
===========================================================
Processing in install mode.
Source root is /opt/oracle.RecoveryAppliance/zdlra
Destination root is /u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms
Backout root is /u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms/install/zdlra/backout_0001
Destination appears to be an installed shiphome.
library = libzdlraserver19.a
Performing object copies for libserver19.a.
…
…
Processing install/zdlra files.
Copying /opt/oracle.RecoveryAppliance/zdlra/install/zdlra/zdlra-software-id to /u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms/install/zdlra/zdlra-software-id
…
…
Wed Jun  3 11:00:39 2020: Start: Update Sys Package.
Wed Jun  3 11:00:39 2020: Switching to UID: 1001, GID: 1002
Wed Jun  3 11:00:39 2020: Set Command '/u01/app/oracle/product/19.0.0.0/dbhome_1/bin/sqlplus       -s / AS SYSDBA <<EOF
…
…
Granting VPD privileges to the owner of the base catalog schema RASYS

========================================
VPD SETUP STATUS:
VPD privileges granted successfully!
Connect to RMAN base catalog and perform UPGRADE CATALOG.
…
…
Wed Jun  3 11:00:51 2020: Set Command '/u01/app/oracle/product/19.0.0.0/dbhome_1/bin/rman catalog /@install.local cmdfile=/opt/oracle.RecoveryAppliance/install/upgrade.rman' timeout to 0.
…
…
Wed Jun  3 11:18:08 2020: Start: Enable RA Backup
Wed Jun  3 11:18:08 2020: Create Symlink succeeded on zeroinsg02.
Wed Jun  3 11:18:08 2020: Create Symlink succeeded on zeroinsg01.
Wed Jun  3 11:18:08 2020: ra_export.pl successfully enabled.
…
…
Wed Jun  3 11:34:14 2020: End: Post Patch Clean Up
Wed Jun  3 11:34:14 2020: Deleted restart flag /opt/oracle.RecoveryAppliance/data/racli_patch_appliance.restart
Wed Jun  3 11:34:14 2020: End: Patch Recovery Appliance - Step [Patch]

As you can see, everything is done by racli, since GI patches, OH metadata, and ZDLRA patches.

06 – Post Patch

As a post patch, we can check again the appliance and verify if everything is running correctly:

[root@zeroinsg01 ~]# racli version
Recovery Appliance Version:
        exadata image: 19.2.3.0.0.190621
        rarpm version: ra_automation-19.2.1.1.1.202003-31304538.x86_64
        rdbms version: RDBMS_19.3.0.0.190416DBRU_LINUX.X64_RELEASE
        transaction  : kadjei_bug-31304538
        zdlra version: ZDLRA_19.2.1.1.1.202003_LINUX.X64_RELEASE
[root@zeroinsg01 ~]# racli status appliance
zeroinsg01 db         state:  [ONLINE]
zeroinsg01 crs        state:  [ONLINE]
zeroinsg01 ra_server  state:  [ONLINE]
zeroinsg02 ra_server  state:  [ONLINE]
zeroinsg02 db         state:  [ONLINE]
zeroinsg02 crs        state:  [ONLINE]
[root@zeroinsg01 ~]#
[root@zeroinsg01 ~]#
[root@zeroinsg01 ~]# racli run check --all
Wed Jun  3 12:16:18 2020: Start: racli run check --all
  Created log file zeroinsg01.bnet.luxds.net:/opt/oracle.RecoveryAppliance/log/racli_run_check_20200603.1216.log

Wed Jun  3 12:16:21 2020: CHECK: RA Services                         - PASS
Wed Jun  3 12:16:31 2020: CHECK: Exadata Image Version               - PASS
Wed Jun  3 12:16:32 2020: CHECK: Active Incidents                    - PASS
Wed Jun  3 12:16:40 2020: CHECK: Init Parameters                     - PASS
Wed Jun  3 12:16:41 2020: CHECK: Invalid Objects                     - PASS
Wed Jun  3 12:16:41 2020: CHECK: Export Backup                       - PASS
Wed Jun  3 12:16:42 2020: CHECK: ZDLRA Rasys Wallet                  - PASS
Wed Jun  3 12:16:46 2020: CHECK: Compute Node AlertHistory
Wed Jun  3 12:16:46 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 12:16:46 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 12:16:53 2020: CHECK: Storage Cell AlertHistory
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm02]                   - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm05]                   - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm01]                   - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm04]                   - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm06]                   - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zerocadm03]                   - PASS
Wed Jun  3 12:16:53 2020: CHECK: Oracle User Password Expires
Wed Jun  3 12:16:53 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 12:16:53 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 12:16:54 2020: CHECK: ZDLRA Version
Wed Jun  3 12:16:54 2020:   HOST: [zeroinsg01]                      - PASS
Wed Jun  3 12:16:54 2020:   HOST: [zeroinsg02]                      - PASS
Wed Jun  3 12:16:54 2020: End: racli run check --all
[root@zeroinsg01 ~]#

If you faced whatever the error, open one SR against Oracle at MOS to receive the correct feedback and fix. Never tries to execute some commands to “fix” because you can corrupt the database and lost data, backups, and your job.

Upgrade and Replication

Since ZDLRA can be used in replicated mode, where the upstream send to downstream, it is important that whoever receives the backup can do that. This means that it is recommended to update always the downstream first. This allows that when receiving the backups from upstream, the internal ZDLRA database and the library can handle it.

Conclusion

The process to update is not complicated, but some details need to be checked before start the process. The patch process is different than the upgrade that I described at my previous post since the major release continued the same. It can be simpler, but the details are very, very, similar.  

 

References

Some references about how to do the patch and handle the issues

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

Leave a Reply

Your email address will not be published. Required fields are marked *