Shrink ASM Diskgroup and Exadata Grid Disks

Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify (and “fix”) if you have something in the ASM internal extent map that can block the shrink.

Here, in this scenario, I will reduce the size for grid disks linked with diskgroup DATAC8 (that run in a cluster for VM #08).

And before continuing, be aware of Exadata disk division:

ASM Extent Map and Moves

So, my actual usage space for DATAC8 is:

ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512             512   4096  4194304  29921280  29790140           498688        14645726              0             Y  DATAC8/
MOUNTED  NORMAL  N         512             512   4096  4194304   6144000   6142020           102400         3019810              0             N  RECOC8/
ASMCMD>

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free
  2  from v$asm_diskgroup
  3  order by 1;

NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE
------------------------------ ---------- ---------- ---------- ----------
DATAC8                           29921280   29790140     131140      99.56
RECOC8                            6144000    6142020       1980      99.97

SQL>

As you can see above, the RAW size is around 29.2 TB (14.6 after the ASM NORMAL mirror) and the actual free space is around 14.3TB. So, plenty of space for the reduction in this case. Unfortunately, even if I have 99% of free space don’t mean that I can reduce instantly to 1TB.

This occurs because, probably, I have a fragmented diskgroup and something can be allocated at the end of some disk and will deny me do reduce the disk. I will not dig in this topic but will show you how to check this. If you want to check with more details you can read these two articles: http://asmsupportguy.blogspot.com/2011/06/asm-file-extent-map.html and http://asmsupportguy.blogspot.com/2012/10/where-is-my-data.html.  

The idea is to use the ASM extend map from X$KFFXP to discover the object that has the MAX AU_KFFXP into some disk (whatever which one). So, I made:

SQL> select GROUP_NUMBER, name from v$asm_diskgroup;

GROUP_NUMBER NAME
------------ ------------------------------
           1 DATAC8
           2 RECOC8

SQL>
SQL> select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=1;

VALUE
--------------------------------------------------------------------------------
4194304

SQL>
SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)
-------------
       114195

SQL>
SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114195;

NUMBER_KFFXP
------------
         262

SQL>
SQL> select name from v$asm_alias where FILE_NUMBER = 262 and GROUP_NUMBER = 1;

NAME
----------------------------------------------------------------------
group_1.262.983716383

SQL>

Above I discovered that the file 252 (that it is a redo – will show after), is allocated in the allocation unit 114195 from diskgroup 1 (that have – and default fore Exadata – allocation unit of 4MB). Doing a little count (114195*4), this means that this file resides in somewhere around 446GB into some disk. And if I try to reduce the disk below this value, I will receive an error.

To solve this we need to move files, in this case, the file is related to MGMTDB:

ASMCMD> ls -l +DATAC8/_MGMTDB/ONLINELOG/
Type       Redund  Striped  Time             Sys  Name
ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_1.262.983716383
ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_2.264.983716383
ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_3.263.983716383
ASMCMD>
ASMCMD> ls -l +DATAC8
Type      Redund  Striped  Time         Sys  Name
                                        Y    ASM/
                                        Y    _MGMTDB/
                                        Y    exa-cl8/
PASSWORD  HIGH    COARSE   JAN 20 2017  N    orapwasm => +DATAC8/ASM/PASSWORD/pwdasm.256.933784119
PASSWORD  HIGH    COARSE   AUG 09 2018  N    orapwasm_backup => +DATAC8/ASM/PASSWORD/pwdasm.1471.983713235
ASMCMD> ls -l +RECOC8
ASMCMD>

To move MGMTDB we can follow the steps from note 2065175.1 and use the script mdbutil.pl. In this case, I moved to RECOC8:

[grid@exa01vm08 +ASM1]$ /tmp/MGMTDB/mdbutil.pl --mvmgmtdb --target=+RECOC8 -debug
mdbutil.pl version : 1.98
2019-06-21 11:43:33: D Executing: /u01/app/18.0.0/grid/bin/srvctl status diskgroup -g RECOC8
2019-06-21 11:43:34: D Exit code: 0
2019-06-21 11:43:34: D Output of last command execution:
Disk Group RECOC8 is running on exa01vm08
2019-06-21 11:43:34: D Executing: /u01/app/18.0.0/grid/bin/srvctl status mgmtdb
2019-06-21 11:43:35: D Exit code: 0
2019-06-21 11:43:35: D Output of last command execution:
…
…
2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/crsctl query crs activeversion
2019-06-21 11:53:20: D Exit code: 0
2019-06-21 11:53:20: D Output of last command execution:
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
2019-06-21 11:53:20: I Starting the Cluster Health Analysis Resource
2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/srvctl start cha
2019-06-21 11:53:22: D Exit code: 0
2019-06-21 11:53:22: D Output of last command execution:
2019-06-21 11:53:22: I MGMTDB Successfully moved to +RECOC8!
[grid@exa01vm08 +ASM1]$

The output above was cropped and you can see in raw here. And after that, doing the same query as above, I checked again the extent map:

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)
-------------
       114132

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114132;

NUMBER_KFFXP
------------
         255

SQL> select name from v$asm_alias where FILE_NUMBER = 255 and GROUP_NUMBER = 1;

NAME
----------------------------------------------------------------------
REGISTRY.255.933784121

SQL>

So, more one file to move and not it is related to OCR. To move I made:

[root@exa01vm08 ~]# export ORACLE_HOME=/u01/app/18.0.0/grid
[root@exa01vm08 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -add +RECOC8
[root@exa01vm08 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      88624
         Available space (kbytes) :     403060
         ID                       :  354072626
         Device/File Name         :    +DATAC8
                                    Device/File integrity check succeeded
         Device/File Name         :    +RECOC8
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -delete +DATAC8
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      88624
         Available space (kbytes) :     403060
         ID                       :  354072626
         Device/File Name         :    +RECOC8
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@exa01vm08 ~]#

Above I: added a new place to OCR store the OCR files(RECOC8) and deleted the other (DATAC8). So, we still have more to move (votedisks, older OCR backups and ASM password file). To move votedisks I made:

[root@exa01vm08 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   077367b255804f0abf804a3a3ca8045d (o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04) [DATAC8]
 2. ONLINE   59ef748a7d2a4f43bffb54a90fe1b1a9 (o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05) [DATAC8]
 3. ONLINE   cbaad30809f94fdcbfe5d60f6529ba63 (o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02) [DATAC8]
Located 3 voting disk(s).
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl replace votedisk +RECOC8
Successful addition of voting disk 081e9e767bc44ff2bff6067229378db5.
Successful addition of voting disk 82649e3309d34fa0bf4fd3c89c93e42f.
Successful addition of voting disk 57f3ec44b44b4fcdbf35e716e13011e9.
Successful deletion of voting disk 077367b255804f0abf804a3a3ca8045d.
Successful deletion of voting disk 59ef748a7d2a4f43bffb54a90fe1b1a9.
Successful deletion of voting disk cbaad30809f94fdcbfe5d60f6529ba63.
Successfully replaced voting disk group with +RECOC8.
CRS-4266: Voting file(s) successfully replaced
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   081e9e767bc44ff2bff6067229378db5 (o/200.200.10.11;200.200.10.12/RECOC8_CD_02_exaceladm04) [RECOC8]
 2. ONLINE   82649e3309d34fa0bf4fd3c89c93e42f (o/200.200.10.9;200.200.10.10/RECOC8_CD_02_exaceladm03) [RECOC8]
 3. ONLINE   57f3ec44b44b4fcdbf35e716e13011e9 (o/200.200.10.13;200.200.10.14/RECOC8_CD_03_exaceladm05) [RECOC8]
Located 3 voting disk(s).
[root@exa01vm08 ~]#
And to backups for OCR:
[root@exa01vm08 ~]# ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+DATAC8].
[root@exa01vm08 ~]# ocrconfig -backuploc +RECOC8
[root@exa01vm08 ~]#
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+RECOC8].
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# ocrconfig -manualbackup

exa01vm08     2019/06/21 12:08:05     +RECOC8:/exa-cl8/OCRBACKUP/backup_20190621_120805.ocr.4722.1011528485     671056737

exa01vm08     2018/08/16 12:38:11     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180816_123811.ocr.1511.984314291     671056737

exa01vm08     2018/08/09 15:56:08     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180809_155608.ocr.1508.983721369     2960767134
[root@exa01vm08 ~]#

After one more backup, I deleted manually the old OCR backups. And to move ASM password:

[grid@exa01vm08 +ASM1]$ asmcmd pwget --asm
+DATAC8/orapwASM
[grid@exa01vm08 +ASM1]$ srvctl config asm -detail
ASM home: <CRS home>
Password file: +DATAC8/orapwASM
Backup of Password file: +DATAC8/orapwASM_backup
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM +RECOC8/orapwASM -f
moving +DATAC8/orapwASM -> +RECOC8/orapwASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ srvctl config asm -detail
ASM home: <CRS home>
Password file: +RECOC8/orapwASM
Backup of Password file: +DATAC8/orapwASM_backup
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM_backup +RECOC8/orapwASM_backup -f
moving +DATAC8/orapwASM_backup -> +RECOC8/orapwASM_backup
[grid@exa01vm08 +ASM1]$
[grid@exa01vm08 +ASM1]$

Maybe you don’t need to move everything, I recommend that after every move you check the extent map for ASM and verify if the value less the minimum size per disk that you want.

Since I changed a lot of things I made a restart for the cluster in both nodes:

[root@exa01vm08 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'exa01vm08'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'exa01vm08'
…
CRS-2673: Attempting to stop 'ora.diskmon' on 'exa01vm08'
CRS-2677: Stop of 'ora.diskmon' on 'exa01vm08' succeeded
[root@exa01vm08 ~]#
[root@exa01vm08 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exa01vm08'
…
…
CRS-2672: Attempting to start 'ora.crsd' on 'exa01vm08'
CRS-2676: Start of 'ora.crsd' on 'exa01vm08' succeeded
[root@exa01vm08 ~]#

And after all of these moves I got the extent map with:

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)
-------------
        15149

SQL>
SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 15149;

NUMBER_KFFXP
------------
           1

SQL>

Basically, this means that something is around 60Gb of some disk. But the link is for a file #1. So, to check what is this you can query inside X$KFFXP:

SQL> select NUMBER_KFFXP "ASM file number",
  2  DECODE (NUMBER_KFFXP,
  3          1, 'File directory'
  4          , 2, 'Disk directory'
  5          , 3, 'Active change directory'
  6          , 4, 'Continuing operations directory'
  7          , 5, 'Template directory'
  8          , 6, 'Alias directory'
  9          , 7, 'ADVM file directory'
 10          , 8, 'Disk free space directory'
 11          , 9, 'Attributes directory'
 12          , 10, 'ASM User directory'
 13          , 11, 'ASM user group directory'
 14          , 12, 'Staleness directory'
 15          , 253, 'spfile for ASM instance'
 16          , 254, 'Stale bit map space registry '
 17          , 255, 'Oracle Cluster Repository registry') "ASM metadata file name",
 18  count(AU_KFFXP) "Allocation units"
 19  from X$KFFXP
 20  where GROUP_KFFXP=1
 21  group by NUMBER_KFFXP
 22  order by 1;

ASM file number ASM metadata file name             Allocation units
--------------- ---------------------------------- ----------------
              1 File directory                                   15
              2 Disk directory                                    3
              3 Active change directory                          69
              4 Continuing operations directory                   6
              5 Template directory                                3
              6 Alias directory                                   3
              8 Disk free space directory                         3
              9 Attributes directory                              3
             12 Staleness directory                               3
             13                                                   3
             16                                                   3

ASM file number ASM metadata file name             Allocation units
--------------- ---------------------------------- ----------------
            120                                                   3
            121                                                 180
            253 spfile for ASM instance                           2
            254 Stale bit map space registry                     18
            255 Oracle Cluster Repository registry               83
            262                                                  99
            321                                                 185
            428                                                 221
            492                                                 305
            551                                                 129
            584                                                 705

ASM file number ASM metadata file name             Allocation units
--------------- ---------------------------------- ----------------
            740                                                1865
           1003                                                 105
           1417                                                   2
           4708                                                   6
           4710                                                   6
           4713                                                   6
           4716                                                   6
           4719                                                   6
           4722                                                   6
           4731                                                  52
           4733                                                  52

ASM file number ASM metadata file name             Allocation units
--------------- ---------------------------------- ----------------
           4736                                                  52
           4739                                                 379
           4742                                                 379
           4744                                                  52
           4747                                                 379
           4750                                                  24
           4753                                                 129
           4756                                                5125
           4759                                                7685
           4762                                                1029
           4765                                                 517

ASM file number ASM metadata file name             Allocation units
--------------- ---------------------------------- ----------------
           4768                                                  52
           4771                                                   4
           4773                                               11819

47 rows selected.

SQL>

In this case, it is a file directory ASM, but I will not try to move because it is even impossible in my case because does not appears in ASMCMD command. Basically, can be internal file/directory that we can’t see. Since I know that the diskgroup is empty and the allocation is related to internal ASM, I continued.

Remember at the end of the procedure, after shrinking in grid disk (that I will show later) to move again these files to original diskgroup. I will not cover this new move here in the post but the steps are the same that I showed before.

Shrink in ASM

The shrinking part start in ASM it is simple and you just need to define the value that you want per disk and reduce them. The most critical part here is to define one size for the disk that it is aligned with 16 MB of the grid disk. Different from when you are increasing the space, here, for shrink, you start to define the size in ASM side and after go to grid disk. And since the grid disk in storage cell round down the value to the nearest 16MB value, you can misalign and have ASM disk with a different value than grid disk (and this will be REALLY BAD). A trick secret is to think directly in GB to avoid the error.

For 16 Mb explanation you can check in the Exadata docs https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-42DA2512-667D-443C-93C5-6E5110DFAE21:

Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.

Since I defined to reduce the DATAC8 to around 1.8TB I defined the disks with 30GB:

SQL> select ((30*12)*5) as sizeGB FROM dual;

    SIZEGB
----------
      1800

SQL>

Remember in the beginning that you need to know your environment? Here I have 12 disks per cell and 5 cells:

SQL> set linesize 250
SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks
  2  from v$asm_disk d, v$asm_diskgroup dg
  3  where d.group_number = dg.group_number
  4  and dg.name IN ('DATAC8')
  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status
  6  order by 1,2,3;

NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS
------ ------------------------------ -------- ------------ ------- ------- ----------
DATAC8 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC8 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC8 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC8 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC8 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12

SQL>

After choosing the size that you want, you need to check if all the grid disk as online. It is not recommended to do the shrink with missing disks or even faulty failgroups. To check both at the same time you can do:

SQL> col path format a100
SQL> select dg.name, d.failgroup, d.path
  2  from v$asm_disk d, v$asm_diskgroup dg
  3  where d.group_number = dg.group_number
  4  and dg.name IN ('DATAC8') and state = 'NORMAL'
  5  order by 1,2,3;

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_00_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_01_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_02_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_03_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_04_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_05_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_06_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_07_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_08_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_09_exaceladm01
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_10_exaceladm01

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_11_exaceladm01
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_00_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_01_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_03_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_04_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_05_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_06_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_07_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_08_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_09_exaceladm02

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_10_exaceladm02
DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_11_exaceladm02
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_00_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_01_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_02_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_03_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_04_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_05_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_06_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_07_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_08_exaceladm03

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_09_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_10_exaceladm03
DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_11_exaceladm03
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_00_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_01_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_03_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_04_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_05_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_06_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_07_exaceladm04

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_08_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_09_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_10_exaceladm04
DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_11_exaceladm04
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_00_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_01_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_03_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_04_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_05_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_06_exaceladm05

NAME                           FAILGROUP                      PATH
------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_07_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_08_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_09_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_10_exaceladm05
DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_11_exaceladm05

60 rows selected.

SQL>

After all the checks 16MB align, missing disks, and online failgroups you can do the resize in ASM:

SQL> alter diskgroup DATAC8 resize all size 30720M rebalance power 1024;

Diskgroup altered.

SQL>

As you can see above, I specified in MB and I made this to show you where the error can occur. If I defined (as an example) disk size as 30700M it will not be aligned to 16MB (30700/16 = 1918,75) and the grid disk will be 30688M and if ASM allocates something at the end of the disk you will corrupt something.

After executing the change in ASM you just continue after having nothing in v$asm_operation. And for ASM now you see the size for your diskgroup:

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free
  2  from v$asm_diskgroup
  3  order by 1;

NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE
------------------------------ ---------- ---------- ---------- ----------
DATAC8                            1843200    1837896       5304      99.71
RECOC8                            6144000    6016048     127952      97.92

SQL>

Shrink for Grid Disk

After shrinking in ASM side, you need to reduce in storage side the grid disk to release the space to celldisk. The procedure is the same for the increase and you use ALTER GRIDDISK to specify the new value.

Just to show, that before the shrink in storage cell I have disk defined for grid disk (487GB) and celldisk (346GB free):

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;
         name:                   DATAC8_CD_00_exaceladm01
         asmDiskGroupName:       DATAC8
         asmDiskName:            DATAC8_CD_00_EXACELADM01
         asmFailGroupName:       EXACELADM01
         availableTo:
         cachedBy:               FD_00_exaceladm01
         cachingPolicy:          default
         cellDisk:               CD_00_exaceladm01
         comment:                "Cluster exa-cl8 diskgroup DATAC8"
         creationTime:           2017-01-20T16:19:29+01:00
         diskType:               HardDisk
         errorCount:             0
         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2
         size:                   487G
         status:                 active

CellCLI>

CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;
         name:                   CD_00_exaceladm01
         comment:
         creationTime:           2016-11-29T10:44:55+01:00
         deviceName:             /dev/sda
         devicePartition:        /dev/sda3
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              346.0625G
         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55
         physicalDisk:           PUUK3V
         size:                   7.1192474365234375T
         status:                 normal

CellCLI>

As before, you have two option: execute manually disk by disk or using the script with dcli. Below you see that I created and called the script (I cropped the output but you can see the RAW execution here):

DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM01 size=30720M;
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM02 size=30720M;
…
…
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM02 size=30720M;
dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM03 size=30720M;
dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM04 size=30720M;
dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM05 size=30720M;
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh
exaceladm01: GridDisk DATAC8_CD_00_exaceladm01 successfully altered
exaceladm02: GridDisk DATAC8_CD_00_exaceladm02 successfully altered
…
…
exaceladm02: GridDisk DATAC8_CD_11_exaceladm02 successfully altered
exaceladm03: GridDisk DATAC8_CD_11_exaceladm03 successfully altered
exaceladm04: GridDisk DATAC8_CD_11_exaceladm04 successfully altered
exaceladm05: GridDisk DATAC8_CD_11_exaceladm05 successfully altered
[DOM0 - root@exadbadm01 tmp]$

Check again that I used the value defined in MB, 30720MB in this case. Again, be careful with 16MB align. After the change I have in storage cell:

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;
         name:                   DATAC8_CD_00_exaceladm01
         asmDiskGroupName:       DATAC8
         asmDiskName:            DATAC8_CD_00_EXACELADM01
         asmFailGroupName:       EXACELADM01
         availableTo:
         cachedBy:               FD_00_exaceladm01
         cachingPolicy:          default
         cellDisk:               CD_00_exaceladm01
         comment:                "Cluster exa-cl8 diskgroup DATAC8"
         creationTime:           2017-01-20T16:19:29+01:00
         diskType:               HardDisk
         errorCount:             0
         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2
         size:                   30G
         status:                 active

CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;
         name:                   CD_00_exaceladm01
         comment:
         creationTime:           2016-11-29T10:44:55+01:00
         deviceName:             /dev/sda
         devicePartition:        /dev/sda3
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              803.0625G
         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55
         physicalDisk:           PUUK3V
         size:                   7.1192474365234375T
         status:                 normal

CellCLI>

Conclusion

Execute the shrink for Exadata is something that you don’t realize every day in your daily tasks. I suppose that not in a quarterly task because today is always “add, add, add”. Besides the increase of space, the shrink is trickier, and you really need to take care for more steps. Maybe you need to move some data, restart the cluster, move data again.

But, the most critical part is the 16MB alignment, more than when you add space. This is important because for shrink you define the value in two places: for ASM and for grid disk. If you choose a bad value, the size for grid disk can differ ASM and you will corrupt something. As I told, if you think directly in GB instead of MB for disk sizes, you are safer because GB is always compatible with 16MB.

 

Reference:

How to Resize Grid Disks in Exadata (Doc ID 2176737.1) – https://support.oracle.com/epmos/faces/DocContentDisplay?id=2176737.1

Resizing Grid Disks – https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-570A0C37-907C-4417-BC93-AC4ABAF7E3AD 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

Share it

Leave a Reply

Your email address will not be published. Required fields are marked *