I already wrote about the issue introduced with GI 19.16 in my previous post (click here to read) where (only at Exadata) more space was allocated/reserved by Oracle to guarantee mirror/rebalance. Fortunately, after some months of discussion, they rollbacked the change and released one patch that can be applied at GI 19.19.

The patch was released on 12 of June and it is the number 35285795 and can be only applied at GI 19.19. But to have your space back again there is one important rule: your mirroring needs to be HIGH. This is necessary because the “Smart Rebalance” that allows your disk to be dropped without losing the mirroring. I will write another post just to talk about it.

Before patch

Look at my server here and you can notice that I am running GI 19.19 and also the values for diskgrouop metadata like the column REQUIRED_MIRROR_FREE_MB from ASMCMD and V$ASM_DISKGROUP:

[grid@exacl01sv01 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
35309390;OCW Interim patch for 35309390
35004974;JDK BUNDLE PATCH 19.0.0.0.230418
35107512;TOMCAT RELEASE UPDATE 19.0.0.0.0 (35107512)
35050325;ACFS RELEASE UPDATE 19.19.0.0.0 (35050325)
35042068;Database Release Update : 19.19.0.0.230418 (35042068)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.
[grid@exacl01sv01 ~]$ asmcmd
ASMCMD>
ASMCMD> lsdg
State    Type  Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  HIGH  N         512             512   4096  4194304  503315712   66454840         45298414         7052142              0             Y  DATAC1/
MOUNTED  HIGH  N         512             512   4096  4194304  125828352  104306364         11324551        30993937              0             N  RECOC1/
ASMCMD> exit
[grid@exacl01sv01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 14 16:54:33 2023
Version 19.19.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0

SQL> set linesize 255
SQL> select NAME, TOTAL_MB,FREE_MB,REQUIRED_MIRROR_FREE_MB, USABLE_FILE_MB from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ---------- ---------- ----------------------- --------------
DATAC1                          503315712   66454840                45298414        7052142
RECOC1                          125828352  104302284                11324551       30992577

SQL>

As you can notice, more than 44TB (RAW) are reserved due to the constraints to sustain diskgroup redundancy/mirroring.

Preparing for Patch

Since this patch change the I/O subsystem for ASM is needed to stop all databases running over the node (the patch is RAC Rolling, so, can be done node by node). If you have a small number of databases you can stop one by one, but another option is to use the “srvctl stop home” to stop all databases linked to that particular Oracle Home.

Patching

Downloading and Checking Conflict can be done previously but now we prepare the patch folder (as GRID user), and check conflicts:

[grid@exacl01sv01 ~]$ mkdir /tmp/patch-gi/
[grid@exacl01sv01 ~]$
[grid@exacl01sv01 ~]$ cd /tmp/patch-gi/
[grid@exacl01sv01 patch-gi]$ cp/scp <FROM_SOMEWHERE> ./
[grid@exacl01sv01 patch-gi]$
[grid@exacl01sv01 patch-gi]$ unzip -q p35285795_1919000DBRU_Linux-x86-64.zip
[grid@exacl01sv01 patch-gi]$
[grid@exacl01sv01 patch-gi]$ ls -l
total 1956
drwxr-xr-x 4 grid oinstall      48 Jun 12 13:23 35285795
-rw-r--r-- 1 grid oinstall 1990874 Jun 14 16:57 p35285795_1919000DBRU_Linux-x86-64.zip
-rw-rw-r-- 1 grid oinstall    4456 Jun 12 22:26 PatchSearch.xml
[grid@exacl01sv01 patch-gi]$
[grid@exacl01sv01 patch-gi]$
[grid@exacl01sv01 patch-gi]$ cd 35285795/
[grid@exacl01sv01 35285795]$
[grid@exacl01sv01 35285795]$
[grid@exacl01sv01 35285795]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.37
Copyright (c) 2023, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/19.0.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.37
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2023-06-14_16-57-33PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.
[grid@exacl01sv01 35285795]$

Prepatch

After stopping the databases and passing the precheck, we need to call the “prepatch” as the root user:

[root@exacl01sv01 ~]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/exacl01sv01/crsconfig/crs_prepatch_apply_inplace_exacl01sv01_2023-06-14_05-07-28PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1458126380].
2023/06/14 17:09:29 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2023/06/14 17:11:04 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2023/06/14 17:11:05 CLSRSC-347: Successfully unlock /u01/app/19.0.0.0/grid
2023/06/14 17:11:08 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
[root@exacl01sv01 ~]#

This step guarantee that everything is released to allow the patch to be applied.

Patching

The patch is a simple opatch with GRID user (no secrets here, a simple opatch apply to specify the OH for the GI and the local where the patch was unzipped):

[root@exacl01sv01 ~]# su - grid
Last login: Wed Jun 14 17:11:19 CEST 2023 from exacl01sv02.sim.on.net.com on ssh
[grid@exacl01sv01 ~]$
[grid@exacl01sv01 ~]$ cd /tmp/patch-gi/35285795/
[grid@exacl01sv01 35285795]$
[grid@exacl01sv01 35285795]$
[grid@exacl01sv01 35285795]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/19.0.0.0/grid -local /tmp/patch-gi/35285795
Oracle Interim Patch Installer version 12.2.0.1.37
Copyright (c) 2023, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/19.0.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.37
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2023-06-14_17-11-59PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   35285795

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/19.0.0.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '35285795' to OH '/u01/app/19.0.0.0/grid'

Patching component oracle.rdbms, 19.0.0.0.0...
Patch 35285795 successfully applied.
Log file location: /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2023-06-14_17-11-59PM_1.log

OPatch succeeded.
[grid@exacl01sv01 35285795]$
[grid@exacl01sv01 35285795]$ $ORACLE_HOME/OPatch/opatch lspatches
35285795;EXADATA  ISSUE INSUFFICIENT SPACE FOR RESTORING REDUNDANCY WITH SMART REBALANCE
35309390;OCW Interim patch for 35309390
35004974;JDK BUNDLE PATCH 19.0.0.0.230418
35107512;TOMCAT RELEASE UPDATE 19.0.0.0.0 (35107512)
35050325;ACFS RELEASE UPDATE 19.19.0.0.0 (35050325)
35042068;Database Release Update : 19.19.0.0.230418 (35042068)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.
[grid@exacl01sv01 35285795]$ logout
[root@exacl01sv01 ~]#

Postpatch

After the patch is applied with success at the node, with the root user we can call the “pospatch” to start the services again:

[root@exacl01sv01 ~]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -postpatch
Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/exacl01sv01/crsconfig/crs_postpatch_apply_inplace_exacl01sv01_2023-06-14_05-14-17PM.log
2023/06/14 17:14:26 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1458126380].
2023/06/14 17:18:07 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2023/06/14 17:18:08 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@exacl01sv01 ~]# 2023/06/14 17:18:38 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
[root@exacl01sv01 ~]#

New Frontier

Now with the patch, we can see a new value for REQUIRED_MIRROR_FREE_MB:

[grid@exacl01sv01 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
35285795;EXADATA  ISSUE INSUFFICIENT SPACE FOR RESTORING REDUNDANCY WITH SMART REBALANCE
35309390;OCW Interim patch for 35309390
35004974;JDK BUNDLE PATCH 19.0.0.0.230418
35107512;TOMCAT RELEASE UPDATE 19.0.0.0.0 (35107512)
35050325;ACFS RELEASE UPDATE 19.19.0.0.0 (35050325)
35042068;Database Release Update : 19.19.0.0.230418 (35042068)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.
[grid@exacl01sv01 ~]$ asmcmd
ASMCMD> lsdg
State    Type  Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  HIGH  N         512             512   4096  4194304  503315712   66452020                0        22150673              0             Y  DATAC1/
MOUNTED  HIGH  N         512             512   4096  4194304  125828352  104103492                0        34701164              0             N  RECOC1/
ASMCMD> exit
[grid@exacl01sv01 ~]$
[grid@exacl01sv01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 14 17:29:39 2023
Version 19.19.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0

SQL> set linesize 255
SQL> select NAME, TOTAL_MB,FREE_MB,REQUIRED_MIRROR_FREE_MB, USABLE_FILE_MB from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ---------- ---------- ----------------------- --------------
DATAC1                          503315712   66449320                       0       22149773
RECOC1                          125828352  104059596                       0       34686532

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[grid@exacl01sv01 ~]$

As you noticed, the value now is 0. This means that (after the patch) nothing is reserved for mirroring. It is even better than before when some space was reserved using this formula (disk size * #redudancy factor – check my previous post).  And the diskgroup received back 44TB (RAW), around 14.5 (already discounted mirror factor), look at the value now for USABLE_FILE_MB.

So, is a quite nice change and it is completely linked with the Smart Rebalance since the only space needed is the redundancy factor of your diskgroup. Resuming a lot (the Smart Rebalance) the space required (for redundancy) is already allocated in partner disks, and in case of failure, the space (needed for rebalance) is already available in the partner disks (of the failed disk). And, when your disk is replaced the new disk is not dropped (as before), it is replaced (with command REPLACE DISK – that I explained before in this post: ASM, REPLACE DISK Command).

But if I don’t have free space? Well, in this case, your rebalance will freeze until replace the failed disk. But don’t try to lose another disk at your diskgroup.

In the next post, I will talk more about the Smart Rebalance and how it works. But to finish, you can start databases at this node and patch the other nodes as well.

Conclusion

The first is simple: Thank You Oracle for listening community/customers and solving the issue in a good way. Well done. Now, what is presented in the datasheet as usable size, can be really used. Don’t forget to update the docs too. If you don’t want to patch, version GI 19.20 (or 19.21) probably will include the patch already.