{"id":736,"date":"2020-06-07T20:34:41","date_gmt":"2020-06-07T23:34:41","guid":{"rendered":"https:\/\/www.fernandosimon.com\/blog\/?p=736"},"modified":"2023-02-01T06:36:23","modified_gmt":"2023-02-01T09:36:23","slug":"zdlra-patch-update-the-recovery-appliance","status":"publish","type":"post","link":"https:\/\/www.fernandosimon.com\/blog\/zdlra-patch-update-the-recovery-appliance\/","title":{"rendered":"ZDLRA, Patch\/Update the Recovery Appliance"},"content":{"rendered":"

The process of patch ZDLRA is not complicated, but it is important to be aware of some details. The most important is from where you are until where you want to go. This is crucial because it will define what commands you will need to execute.<\/p>\n

If you read the previous post<\/a> about the process, you can notice that I was running the ZDLRA 12.2 version, and forwarded to 19.2 version. In that case, I needed to use the upgrade path since I was changing the major release and the racli commands had the \u201cupgrade\u201d parameter.<\/p>\n

In this post I will show how to do a simple update (or patch apply) for ZDLRA, this means that I will remain inside the same major release for recovery appliance library. Some steps and checks are the same.<\/p>\n

Whatever you need to do (patch or upgrade), the startup point it is the note 1927416.1<\/a> that cover the supported versions for ZDLRA. There it is possible to find all the supported versions for the recovery appliance library as well as the Exadata versions. Please, not upgrade the Exadata stack with a version that is not listed on this page.<\/p>\n

<\/p>\n

Where we are<\/h2>\n

The first step is always to discover what version for ZDLRA and Exadata Stack you are running. The simple way to do that is using the command \u201cracli version<\/em>\u201d:<\/p>\n

[root@zeroinsg01 ~]# racli version\r\nRecovery Appliance Version:\r\n        exadata image: 19.2.3.0.0.190621\r\n        rarpm version: ra_automation-19.2.1.1.1.202001-31014797.x86_64\r\n        rdbms version: RDBMS_19.3.0.0.190416DBRU_LINUX.X64_RELEASE\r\n        transaction  : kadjei_bug-31014797\r\n        zdlra version: ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE\r\n[root@zeroinsg01 ~]#<\/pre>\n
The way to read this is that: rpm version<\/em> is the running version, zdlra version<\/em> is the base release (202001). The major release is 19.2.1.1.1. So, it is possible to discover that the currently running version for ZDLRA is 19.2.1.1.1.202001-31014797 and the Exadata image is 19.2.3.0.0.190621.<\/p>\n
Where we want to go<\/h2>\n
As I wrote before, the startup point it is the note 1927416.1<\/a>. There we can check the available versions for ZDLRA, and in this example, I will patch to version 19.2.1.1.1-202003-31304538 (released 18-May-2020).<\/p>\n
So, I will jump from 19.2.1.1.1.202001-31014797 to 19.2.1.1.1-202003-31304538, but I will remain inside of the same major release of ZDLRA, the 19.2.1.1.1. I will jump from base release (like PSU for OH) from 202001 (released in January) to 202003 (released in March).<\/p>\n
Exadata Supported Versions<\/h3>\n
Again, checking the note 1927416.1<\/a> that covers the supported versions for ZDLRA is possible to check the compatibility matrix for Exadata System Software (database and storage servers) and discover if it is needed to upgrade\/patch the Exadata stack.<\/p>\n
\"\"<\/a><\/p>\n
Look above that the current version for Exadata is supported. So, this means that I can do just the patch apply for ZDLRA. If you want to update the Exadata Stack you can check my post on how to do that<\/a>, but the process is very, very, similar for Exadata Patch apply.<\/p>\n
The only detail is that usually, we update the ZDLRA library first, and after the Exadata Stack. This occurs, (because as you can see in the image above), that the current version that I am running (19.2.1.1.1.202001-31014797) is not compatible with the Exadata 19.3.7.0.0. So, first, update ZDLRA, and after Exadata stack. In this example, I will just update the ZDLRA library.<\/p>\n
Patch Process<\/h2>\n
Download<\/h3>\n
Since we will continue in the same major release of ZDLRA (19.2.1.1.1), this means that the OH and GI will continue at the same release and will not suffer any kind of version upgrade (like 12 to 19). So, we don\u2019t need to download additional patches or binaries. In my previous post<\/a>, I already explained this.<\/p>\n
As you know, the ZDLRA library contains the updates for the OH and GI, so the compatible PSU\u2019s will be applied and they are inside of downloaded patched:<\/p>\n
\"\"<\/a><\/p>\n
Where put the files<\/h3>\n
As I wrote in my previous post<\/strong>, all the files for ZDLRA need to be stored at \/radump at the database server node 01.<\/strong> It is fixed in the procedure and it is a requirement.<\/p>\n
And as best practices, I recommend that before copy the new files, remove all older patches from \/radump in both nodes<\/strong>. And this include files that are inside the ZDLRA patch (like ra_init_param_check.pl, load_init_param.sh, load_init_param.pl,  dbmsrsadm.sql, dbmsrsadmpreq.sql, prvtrsadm.sql, ra_preinstall.pl).<\/p>\n
01 – Precheck<\/h3>\n
The first step is to do a simple precheck of the ZDLRA stack. The simple and fast way is executing the \u201cracli status appliance<\/em>\u201d:<\/p>\n
[root@zeroinsg01 ~]# racli status appliance\r\nzeroinsg01 crs        state:  [ONLINE]\r\nzeroinsg01 db         state:  [ONLINE]\r\nzeroinsg01 ra_server  state:  [ONLINE]\r\nzeroinsg02 crs        state:  [ONLINE]\r\nzeroinsg02 ra_server  state:  [ONLINE]\r\nzeroinsg02 db         state:  [ONLINE]\r\n[root@zeroinsg01 ~]#<\/pre>\n
And the \u201cracli run check –all<\/em>“:<\/p>\n
[root@zeroinsg01 ~]# racli run check --all\r\nWed Jun  3 10:07:21 2020: Start: racli run check --all\r\n  Created log file zeroinsg01:\/opt\/oracle.RecoveryAppliance\/log\/racli_run_check_20200603.1007.log\r\n\r\nWed Jun  3 10:07:26 2020: CHECK: RA Services                         - PASS\r\nWed Jun  3 10:07:37 2020: CHECK: Exadata Image Version               - PASS\r\nWed Jun  3 10:07:38 2020: CHECK: Active Incidents                    - PASS\r\nWed Jun  3 10:07:46 2020: CHECK: Init Parameters                     - PASS\r\nWed Jun  3 10:07:47 2020: CHECK: Invalid Objects                     - PASS\r\nWed Jun  3 10:07:48 2020: CHECK: Export Backup                       - PASS\r\nWed Jun  3 10:07:48 2020: CHECK: ZDLRA Rasys Wallet                  - PASS\r\nWed Jun  3 10:07:51 2020: CHECK: Compute Node AlertHistory\r\nWed Jun  3 10:07:51 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 10:07:51 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 10:07:59 2020: CHECK: Storage Cell AlertHistory\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm04]                   - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm02]                   - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm06]                   - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm05]                   - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm01]                   - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zerocadm03]                   - PASS\r\nWed Jun  3 10:07:59 2020: CHECK: Oracle User Password Expires\r\nWed Jun  3 10:07:59 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 10:07:59 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 10:08:00 2020: CHECK: ZDLRA Version\r\nWed Jun  3 10:08:00 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 10:08:00 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 10:08:00 2020: End: racli run check --all\r\n[root@zeroinsg01 ~]#<\/pre>\n
With that, every error can be checked, reported, and fixed before continuing the process.<\/p>\n
02 \u2013 Unzip Library<\/h3>\n
The second step is, as oracle user<\/strong>, unzip the ZDLRA patch. This is done inside of \/radump folder at node 01:<\/p>\n
[oracle@zeroinsg01 radump]$ unzip p31304538_192111_Linux-x86-64.zip\r\nArchive:  p31304538_192111_Linux-x86-64.zip\r\n  inflating: set_env.sh\r\n extracting: p6880880_180000_Linux-x86-64.zip\r\n  inflating: dbmsrsadmpreq.sql\r\n  inflating: dbmsrsadm.sql\r\n  inflating: prvtrsadm.sql\r\n  inflating: create_raoratab.pl\r\n  inflating: create_raoratab.sh\r\n  inflating: get_versions.pl\r\n  inflating: get_versions.sh\r\n  inflating: ra_init_param_check.pl\r\n  inflating: ra_init_param_check.sh\r\n  inflating: ra_precheck.pl\r\n  inflating: ra_precheck.sh\r\n  inflating: ra_preinstall.pl\r\n  inflating: ra_automation-19.2.1.1.1.202003-31304538.x86_64.rpm\r\n  inflating: run_set_env.sh\r\n extracting: p29908639_193000DBRU_Linux-x86-64.zip\r\n extracting: p29232533_190000_Linux-x86-64.zip\r\n extracting: p29726449_190000_Linux-x86-64.zip\r\n extracting: p29391849_194000DBRU_Linux-x86-64.zip\r\n extracting: p30177140_194000DBRU_Linux-x86-64.zip\r\n extracting: p30540969_190000_Linux-x86-64.zip\r\n extracting: p30312546_190000_Linux-x86-64.zip\r\n extracting: p30458593_190000_Linux-x86-64.zip\r\n extracting: p30363621_190000_Linux-x86-64.zip\r\n extracting: p28538439_190000_Linux-x86-64.zip\r\n extracting: p30094929_190000_Linux-x86-64.zip\r\n extracting: p29708769_190000_Linux-x86-64.zip\r\n  inflating: README.txt\r\n[oracle@zeroinsg01 radump]$<\/pre>\n
As you can see the patch p31304538_192111_Linux-x86-64.zip contains inside the PSU\/One-offs (p* files) patches for OH and GI.<\/p>\n
03 \u2013 ra_preinstall.pl<\/h3>\n
The third is step is to execute the ra_preinstall.pl, as root user<\/strong>, to start the process. This step removes the old ZDLRA rpm library from the system and installs the new one (in both nodes). Besides that, execute some checks as well.<\/p>\n
As explained in my previous post, if you have some open incidents you can clean it (the process is described there), but also is not recommended to restart this ZDLRA after this moment because the ZDLRA binaries were changed (rpm was changed) but the database itself was not upgraded. Of course, that library supposes to be\/have interoperability, but if it is possible to avoid the problem, it is desired.<\/p>\n
[root@zeroinsg01 ~]# cd \/radump\/\r\n[root@zeroinsg01 radump]#\r\n[root@zeroinsg01 radump]# which perl\r\n\/opt\/oracle.RecoveryAppliance\/bin\/perl\r\n[root@zeroinsg01 radump]#\r\n[root@zeroinsg01 radump]# perl ra_preinstall.pl\r\nStart: Running ra_preinstall.pl on zeroinsg01.\r\nNOTE:\r\n    Current deployed RPM [ra_automation-19.2.1.1.1.202001-31014797.x86_64.rpm] not found!\r\n    If you continue without an old RPM, rollback will not be possible.\r\n    Refer to the README.txt included in this ZDLRA Patch for more details.\r\n\r\n    Do you want to continue? (y|n):  y\r\nNote:\r\n    The ra_preinstall.pl manages the ra_automation RPM,\r\n    and provides a --rollback option.\r\n    The RPM is updated during ra preinstall. Rollback is feasible if the old rpm is found.\r\n    You do not need to update RPM separately.\r\n\r\n    Refer to the README.txt included in this ZDLRA Patch for more details.\r\n\r\n    Do you want to continue (y|n):  y\r\nDeployed RPM:  ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE\r\nInstalled RPM: ZDLRA_19.2.1.1.1.202001_LINUX.X64_RELEASE\r\nRPM matches.\r\nFuse group already exists. Skipping.\r\nStart Update sshd_config\r\nEnd Update sshd_config\r\nEnd: Running ra_preinstall.pl on zeroinsg01.\r\nCopying ra_preinstall.pl, create_raoratab.* and new RPM to remote node zeroinsg02. Password may be required to connect.\r\nra_preinstall.pl                                             100%   42KB  62.5MB\/s   00:00\r\ncreate_raoratab.pl                                           100% 3251    13.7MB\/s   00:00\r\ncreate_raoratab.sh                                           100%  991     5.5MB\/s   00:00\r\nrun_set_env.sh                                               100% 1000     4.4MB\/s   00:00\r\nset_env.sh                                                   100% 2420    13.5MB\/s   00:00\r\nra_automation-19.2.1.1.1.202003-31304538.x86_64.rpm          100% 1903MB 227.2MB\/s   00:08\r\nCreated log \/opt\/oracle.RecoveryAppliance\/log\/create_raoratab.log\r\nStart: Running ra_preinstall.pl on zeroinsg02.\r\nFuse group already exists. Skipping.\r\nStart Update sshd_config\r\nEnd Update sshd_config\r\nEnd: Running ra_preinstall.pl on zeroinsg02.\r\nStart Restart sshd\r\nEnd Restart sshd\r\nStart: Check Init Parameters\r\nCreated log file \/opt\/oracle.RecoveryAppliance\/log\/ra_init_param_check.log\r\nAll init parameters have been validated.\r\nEnd: Check Init Parameters\r\nStart: Remove current RPM and install new RPM on zeroinsg01.\r\nEnd: Remove current RPM and install new RPM on zeroinsg01.\r\nStart: Remove current RPM and install new RPM on zeroinsg02.\r\nEnd: Remove current RPM and install new RPM on zeroinsg02.\r\nCreated log \/opt\/oracle.RecoveryAppliance\/log\/raprecheck.log\r\nStart: Run Pre check for Patch\r\nWed Jun  3 10:10:34 2020: Start: Check ZDLRA Services\r\nWed Jun  3 10:10:38 2020: End: Check ZDLRA Services\r\nWed Jun  3 10:10:38 2020: Start: Check ASM rebalance\r\nWed Jun  3 10:10:39 2020: End: Check ASM rebalance\r\nWed Jun  3 10:10:39 2020: Start: Check Cluster\r\nWed Jun  3 10:11:48 2020: End: Check Cluster\r\nWed Jun  3 10:11:53 2020: Start: Check Open Incidents\r\nWed Jun  3 10:11:54 2020: End: Check Open Incidents\r\nWed Jun  3 10:11:54 2020: Start: Check Invalid Objects\r\nWed Jun  3 10:11:58 2020: End: Check Invalid Objects\r\nWed Jun  3 10:11:58 2020: Start: Check Init Parameters\r\nWed Jun  3 10:11:59 2020: End: Check Init Parameters\r\nWed Jun  3 10:11:59 2020: Start: Check compute node oracle access\r\nWed Jun  3 10:11:59 2020: End: Check compute node oracle access\r\nWed Jun  3 10:11:59 2020: Start: Check TFA\/AHF status\r\nWed Jun  3 10:12:03 2020: End: Check TFA\/AHF status\r\nEnd: Run Pre check for Patch\r\nStart Restart sshd\r\n\r\n\r\n!!!NOTE!!!\r\n\r\nExit and log back into [zeroinsg01] prior to continuing\r\n\r\n\r\n!!!NOTE!!!\r\n\r\nEnd Restart sshd\r\n[root@zeroinsg01 radump]#\r\n\r\n\r\n\r\n\r\n\r\nLast login: Wed Jun  3 10:12:30 2020 from \r\n[root@zeroinsg01 ~]#<\/pre>\n
I always check where Perl is coming, be sure that it is from the ZDLRA folder. The warning about rpm file not found means that the current rpm file was not found at \/radump (probably cleaned in some moment), but since I have it in local NFS, I can just copy it if needed.<\/p>\n
04 \u2013 patch appliance –step=1<\/h3>\n
After the rpm is changed in both nodes, we can call the racli to do continue with the patch. The first step is just a precheck phase. This is executed as root:<\/p>\n
[root@zeroinsg01 ~]# \/opt\/oracle.RecoveryAppliance\/bin\/racli patch appliance --step=1\r\nCreated log \/opt\/oracle.RecoveryAppliance\/log\/racli_patch_appliance.log\r\nWed Jun  3 10:13:36 2020: Start: Patch Recovery Appliance - Step [PreCheck]\r\nWed Jun  3 10:13:36 2020: Start: Check ZDLRA Services\r\nWed Jun  3 10:13:40 2020: End: Check ZDLRA Services\r\nWed Jun  3 10:13:40 2020: Start: Check ASM rebalance\r\nWed Jun  3 10:13:41 2020: End: Check ASM rebalance\r\nWed Jun  3 10:13:41 2020: Start: Check Cluster\r\nWed Jun  3 10:14:57 2020: End: Check Cluster\r\nWed Jun  3 10:15:00 2020: Start: Check Open Incidents\r\nWed Jun  3 10:15:01 2020: End: Check Open Incidents\r\nWed Jun  3 10:15:01 2020: Start: Check Invalid Objects\r\nWed Jun  3 10:15:05 2020: End: Check Invalid Objects\r\nWed Jun  3 10:15:05 2020: Start: Check Init Parameters\r\nWed Jun  3 10:15:07 2020: End: Check Init Parameters\r\nWed Jun  3 10:15:07 2020: Start: Check compute node oracle access\r\nWed Jun  3 10:15:07 2020: End: Check compute node oracle access\r\nWed Jun  3 10:15:07 2020: Start: Check TFA\/AHF status\r\nWed Jun  3 10:15:10 2020: End: Check TFA\/AHF status\r\nWed Jun  3 10:15:13 2020: Start: Check RA Patch Levels\r\nWed Jun  3 10:15:13 2020: End: Check RA Patch Levels\r\nWed Jun  3 10:15:13 2020: Start: Check Patches\r\nWed Jun  3 10:15:25 2020: End: Check Patches\r\nWed Jun  3 10:15:25 2020: End: Patch Recovery Appliance - Step [PreCheck]\r\n[root@zeroinsg01 ~]#<\/pre>\n
This check if some errors exist inside ASM or ZDLRA database metadata.<\/p>\n
As you can above, the command racli now uses the parameter \u201cpatch appliance<\/strong>\u201d. In my previous posts, I used \u201cupgrade appliance\u201d. The patch, as explained, is used when we are inside of the same major release.<\/p>\n
05 \u2013 patch appliance –step=2<\/h3>\n
The next step is the patch itself, the racli called as root will apply all patches needed.<\/p>\n
[root@zeroinsg01 ~]# \/opt\/oracle.RecoveryAppliance\/bin\/racli patch appliance --step=2\r\nCreated log \/opt\/oracle.RecoveryAppliance\/log\/racli_patch_appliance.log\r\nWed Jun  3 10:17:46 2020: Start: Patch Recovery Appliance - Step [Patch]\r\nWed Jun  3 10:17:46 2020: Skip: Patch Recovery Appliance - Step [PreCheck] [Already Run]\r\nWed Jun  3 10:17:48 2020: Start: Pre Patch Release Steps\r\nWed Jun  3 10:17:48 2020: End: Pre Patch Release Steps\r\nWed Jun  3 10:17:48 2020: Start: Unpack Patches\r\nWed Jun  3 10:19:24 2020: End: Unpack Patches\r\nWed Jun  3 10:19:24 2020: Start: Rasys configuration\r\nWed Jun  3 10:19:27 2020: End: Rasys configuration\r\nWed Jun  3 10:19:27 2020: Start: Stop ZDLRA Services\r\nWed Jun  3 10:23:26 2020: End: Stop ZDLRA Services\r\nWed Jun  3 10:23:26 2020: Start: Disable ZDLRA Exports\r\nWed Jun  3 10:23:26 2020: End: Disable ZDLRA Exports\r\nWed Jun  3 10:23:26 2020: Start: Check Active Programs\r\nWed Jun  3 10:23:57 2020: Start: Stop Clusterware\r\nWed Jun  3 10:24:34 2020: End: Stop Clusterware\r\nWed Jun  3 10:24:34 2020: Start: Checking active programs\r\nWed Jun  3 10:24:46 2020: End: Checking active programs\r\nWed Jun  3 10:24:46 2020: Start: Start Clusterware\r\nWed Jun  3 10:30:27 2020: End: Start Clusterware\r\nWed Jun  3 10:30:27 2020: End: Check Active Programs\r\nWed Jun  3 10:30:27 2020: Start: Patch OPatch\r\nWed Jun  3 10:30:29 2020: End: Patch OPatch\r\nWed Jun  3 10:30:29 2020: Start: CRS start\r\nWed Jun  3 10:32:56 2020: End: CRS start\r\nWed Jun  3 10:32:56 2020: Start: Installed Patch Check\r\nWed Jun  3 10:35:40 2020: End: Installed Patch Check\r\nWed Jun  3 10:35:40 2020: Start: Apply Bundle Patch\r\nWed Jun  3 10:36:52 2020:       Skipped: Bundle Patch is Current.\r\nWed Jun  3 10:36:52 2020: End: Apply Bundle Patch\r\nWed Jun  3 10:36:53 2020: End: Apply Bundle Patch\r\nWed Jun  3 10:36:53 2020: Start: Apply GI Patches\r\nWed Jun  3 10:54:56 2020: End: Apply GI Patches\r\nWed Jun  3 10:54:56 2020: Start: Install ZDLRA Patch\r\nWed Jun  3 11:00:16 2020: End: Install ZDLRA Patch\r\nWed Jun  3 11:00:16 2020: Start: Apply Datapatch\r\nWed Jun  3 11:00:37 2020: End: Apply Datapatch\r\nWed Jun  3 11:00:37 2020: Start: Rasys Update\r\nWed Jun  3 11:02:37 2020: End: Rasys Update\r\nWed Jun  3 11:02:37 2020: Start: ZDLRA OS Updates\r\nWed Jun  3 11:02:38 2020: End: ZDLRA OS Updates\r\nWed Jun  3 11:02:38 2020: Start: Exachk Update\r\nWed Jun  3 11:02:40 2020: End: Exachk Update\r\nWed Jun  3 11:02:40 2020: Start: AHF Update\r\nWed Jun  3 11:09:24 2020: End: AHF Update\r\nWed Jun  3 11:09:28 2020: Start: Secure Backup Update\r\nWed Jun  3 11:09:28 2020: Start: Tape Update\r\nWed Jun  3 11:09:58 2020: End: Tape Update\r\nWed Jun  3 11:09:58 2020: End: Secure Backup Update\r\nWed Jun  3 11:09:58 2020: Start: Post Patch Service Enable\r\nWed Jun  3 11:10:01 2020: Start: Stop ZDLRA Stack\r\nWed Jun  3 11:13:36 2020: End: Stop ZDLRA Stack\r\nWed Jun  3 11:13:36 2020: Start: Enable ZDLRA\r\nWed Jun  3 11:13:37 2020: End: Enable ZDLRA\r\nWed Jun  3 11:18:08 2020: Start: Enable RA Backup\r\nWed Jun  3 11:18:08 2020: End: Enable RA Backup\r\nWed Jun  3 11:18:08 2020: End: Post Patch Service Enable\r\nWed Jun  3 11:18:08 2020: Start: Reset RA user password\r\nWed Jun  3 11:18:09 2020: End: Reset RA user password\r\nWed Jun  3 11:18:09 2020: Start: Check for EM Agents\r\nWed Jun  3 11:19:00 2020: End: Check for EM Agents\r\nWed Jun  3 11:19:00 2020: Start: Check Init Parameters\r\nWed Jun  3 11:19:00 2020: End: Check Init Parameters\r\nWed Jun  3 11:19:00 2020: Start: Setup OS\r\nWed Jun  3 11:19:35 2020: End: Setup OS\r\nWed Jun  3 11:19:35 2020: Start: Post Patch Clean Up\r\nWed Jun  3 11:34:14 2020: End: Post Patch Clean Up\r\nWed Jun  3 11:34:14 2020: End: Patch Recovery Appliance - Step [Patch]\r\n[root@zeroinsg01 ~]#<\/pre>\n
The patch will apply everything that is needed: OH Patches, GI Patches, OSB, and ZDLRA stack. In this process, the ZDLRA will be \u201coffline\u201d and not receive ingested backups. I recommend not even try to send backups (or archivelogs) to ZDLRA to avoid concurrencies inside rman catalog (since the patch process touches it too).<\/p>\n
You can see below some parts of the patch apply process:<\/p>\n
Wed Jun  3 10:17:48 2020: Got \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1 patch status\r\nWed Jun  3 10:17:48 2020: Checking for 29517242.\r\nWed Jun  3 10:17:48 2020: Found 29517242.\r\nWed Jun  3 10:17:48 2020: Start: Pre Patch Release Steps\r\nWed Jun  3 10:17:48 2020: End: RunLevel 1001\r\nWed Jun  3 10:17:48 2020: Start: RunLevel 1002\r\nWed Jun  3 10:17:48 2020: End: Pre Patch Release Steps\r\nWed Jun  3 10:17:48 2020: Start: Unpack Patches\r\nWed Jun  3 10:17:48 2020: Start: Unpack Patches\r\nWed Jun  3 10:17:48 2020: Checking for patch [29726449]\r\nWed Jun  3 10:17:48 2020: Found patch [29726449]. Checking 29726449.\r\nWed Jun  3 10:17:48 2020: Patch [29726449] md5sum matches. Unpacking 29726449.\r\nWed Jun  3 10:17:48 2020: Switching to UID: 1001, GID: 1002\r\nWed Jun  3 10:17:48 2020: Set Command '\/usr\/bin\/unzip -nuq \/radump\/\/p29726449_190000_Linux-x86-64.zip -d \/\/radump\/\/patch\/\/general' timeout to 900.\r\n\r\nWed Jun  3 10:17:48 2020: Unpacking 29726449 complete. OK\r\nWed Jun  3 10:17:48 2020: Checking for patch [29232533]\r\nWed Jun  3 10:17:48 2020: Found patch [29232533]. Checking 29232533.\r\nWed Jun  3 10:17:48 2020: Patch [29232533] md5sum matches. Unpacking 29232533.\r\n\u2026\r\n\u2026\r\nWed Jun  3 10:24:00 2020: spawn \/usr\/bin\/ssh -o ConnectTimeout=20 -o LogLevel=error -l root zeroinsg02 \/u01\/app\/19.0.0.0\/grid\/bin\/crsctl stop crs -f\r\n\u2026\r\n\u2026\r\nWed Jun  3 10:33:16 2020: Found 28279612.\r\nWed Jun  3 10:33:16 2020: Removing 28279612\r\nWed Jun  3 10:33:16 2020: Set Command '\/usr\/bin\/expect ' timeout to 305.\r\n\r\nWed Jun  3 10:33:43 2020: spawn \/usr\/bin\/ssh -o ConnectTimeout=20 -o LogLevel=error -l oracle zeroinsg02 \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/OPatch\/opatch rollback -id 28279612 -local_node -no_relink -oh \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1 -silent\r\nOracle Interim Patch Installer version 12.2.0.1.17\r\n...\r\n...\r\nWed Jun  3 10:36:52 2020: spawn \/usr\/bin\/ssh -o ConnectTimeout=20 -o LogLevel=error -l root zeroinsg01 \/u01\/app\/19.0.0.0\/grid\/OPatch\/opatchauto apply \/\/radump\/\/patch\/\/gi_opatchauto\/29708769 -oh \/u01\/app\/19.0.0.0\/grid\r\n\r\nOPatchauto session is initiated at Wed Jun  3 10:36:21 2020\r\n\u2026\r\n\u2026\r\n===========================================================\r\ninstall-zdlra-patch starting\r\n===========================================================\r\nProcessing in install mode.\r\nSource root is \/opt\/oracle.RecoveryAppliance\/zdlra\r\nDestination root is \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/rdbms\r\nBackout root is \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/rdbms\/install\/zdlra\/backout_0001\r\nDestination appears to be an installed shiphome.\r\nlibrary = libzdlraserver19.a\r\nPerforming object copies for libserver19.a.\r\n\u2026\r\n\u2026\r\nProcessing install\/zdlra files.\r\nCopying \/opt\/oracle.RecoveryAppliance\/zdlra\/install\/zdlra\/zdlra-software-id to \/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/rdbms\/install\/zdlra\/zdlra-software-id\r\n\u2026\r\n\u2026\r\nWed Jun  3 11:00:39 2020: Start: Update Sys Package.\r\nWed Jun  3 11:00:39 2020: Switching to UID: 1001, GID: 1002\r\nWed Jun  3 11:00:39 2020: Set Command '\/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/bin\/sqlplus       -s \/ AS SYSDBA <<EOF\r\n\u2026\r\n\u2026\r\nGranting VPD privileges to the owner of the base catalog schema RASYS\r\n\r\n========================================\r\nVPD SETUP STATUS:\r\nVPD privileges granted successfully!\r\nConnect to RMAN base catalog and perform UPGRADE CATALOG.\r\n\u2026\r\n\u2026\r\nWed Jun  3 11:00:51 2020: Set Command '\/u01\/app\/oracle\/product\/19.0.0.0\/dbhome_1\/bin\/rman catalog \/@install.local cmdfile=\/opt\/oracle.RecoveryAppliance\/install\/upgrade.rman' timeout to 0.\r\n\u2026\r\n\u2026\r\nWed Jun  3 11:18:08 2020: Start: Enable RA Backup\r\nWed Jun  3 11:18:08 2020: Create Symlink succeeded on zeroinsg02.\r\nWed Jun  3 11:18:08 2020: Create Symlink succeeded on zeroinsg01.\r\nWed Jun  3 11:18:08 2020: ra_export.pl successfully enabled.\r\n\u2026\r\n\u2026\r\nWed Jun  3 11:34:14 2020: End: Post Patch Clean Up\r\nWed Jun  3 11:34:14 2020: Deleted restart flag \/opt\/oracle.RecoveryAppliance\/data\/racli_patch_appliance.restart\r\nWed Jun  3 11:34:14 2020: End: Patch Recovery Appliance - Step [Patch]<\/pre>\n
As you can see, everything is done by racli, since GI patches, OH metadata, and ZDLRA patches.<\/p>\n
06 \u2013 Post Patch<\/h3>\n
As a post patch, we can check again the appliance and verify if everything is running correctly:<\/p>\n
[root@zeroinsg01 ~]# racli version\r\nRecovery Appliance Version:\r\n        exadata image: 19.2.3.0.0.190621\r\n        rarpm version: ra_automation-19.2.1.1.1.202003-31304538.x86_64\r\n        rdbms version: RDBMS_19.3.0.0.190416DBRU_LINUX.X64_RELEASE\r\n        transaction  : kadjei_bug-31304538\r\n        zdlra version: ZDLRA_19.2.1.1.1.202003_LINUX.X64_RELEASE\r\n[root@zeroinsg01 ~]# racli status appliance\r\nzeroinsg01 db         state:  [ONLINE]\r\nzeroinsg01 crs        state:  [ONLINE]\r\nzeroinsg01 ra_server  state:  [ONLINE]\r\nzeroinsg02 ra_server  state:  [ONLINE]\r\nzeroinsg02 db         state:  [ONLINE]\r\nzeroinsg02 crs        state:  [ONLINE]\r\n[root@zeroinsg01 ~]#\r\n[root@zeroinsg01 ~]#\r\n[root@zeroinsg01 ~]# racli run check --all\r\nWed Jun  3 12:16:18 2020: Start: racli run check --all\r\n  Created log file zeroinsg01:\/opt\/oracle.RecoveryAppliance\/log\/racli_run_check_20200603.1216.log\r\n\r\nWed Jun  3 12:16:21 2020: CHECK: RA Services                         - PASS\r\nWed Jun  3 12:16:31 2020: CHECK: Exadata Image Version               - PASS\r\nWed Jun  3 12:16:32 2020: CHECK: Active Incidents                    - PASS\r\nWed Jun  3 12:16:40 2020: CHECK: Init Parameters                     - PASS\r\nWed Jun  3 12:16:41 2020: CHECK: Invalid Objects                     - PASS\r\nWed Jun  3 12:16:41 2020: CHECK: Export Backup                       - PASS\r\nWed Jun  3 12:16:42 2020: CHECK: ZDLRA Rasys Wallet                  - PASS\r\nWed Jun  3 12:16:46 2020: CHECK: Compute Node AlertHistory\r\nWed Jun  3 12:16:46 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 12:16:46 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 12:16:53 2020: CHECK: Storage Cell AlertHistory\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm02]                   - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm05]                   - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm01]                   - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm04]                   - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm06]                   - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zerocadm03]                   - PASS\r\nWed Jun  3 12:16:53 2020: CHECK: Oracle User Password Expires\r\nWed Jun  3 12:16:53 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 12:16:53 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 12:16:54 2020: CHECK: ZDLRA Version\r\nWed Jun  3 12:16:54 2020:   HOST: [zeroinsg01]                      - PASS\r\nWed Jun  3 12:16:54 2020:   HOST: [zeroinsg02]                      - PASS\r\nWed Jun  3 12:16:54 2020: End: racli run check --all\r\n[root@zeroinsg01 ~]#<\/pre>\n
If you faced whatever the error, open one SR against Oracle at MOS to receive the correct feedback and fix. Never tries to execute some commands to \u201cfix\u201d because you can corrupt the database and lost data, backups, and your job.<\/p>\n
Upgrade and Replication<\/h2>\n
Since ZDLRA can be used in replicated mode, where the upstream send to downstream, it is important that whoever receives the backup can do that. This means that it is recommended to update always the downstream first. This allows that when receiving the backups from upstream, the internal ZDLRA database and the library can handle it.<\/p>\n
Conclusion<\/h2>\n
The process to update is not complicated, but some details need to be checked before start the process. The patch process is different than the upgrade that I described at my previous post<\/a> since the major release continued the same. It can be simpler, but the details are very, very, similar.  <\/p>\n
 <\/p>\n
References<\/h2>\n
Some references about how to do the patch and handle the issues<\/p>\n