Increase size for Exadata Grid Disks

A quick article about a maintenance task for Oracle Exadata when you are using OVM and you divided your storage cell disks for every VM. Here I will show you how to extend your Grid Disks to add more space in your ASM diskgroup.

The first thing is being aware of your environment, before everything you need to know the points below because, they are important to calculate the new space, and to avoid do something wrong:

  • Number of cells in your appliance.
  • Number of disks for each cell.
  • Mirroring for your ASM.
  • The VM that you want to add the space.

The “normal” Exadata storage cell has 12 disks, the Extreme Flash version uses 8 disks per storage. If you have doubt about how many disks you have per storage cell, you can connect in each one and check the number of celldisks you have. And before continuing, be aware of Exadata disk division:

To do this change we execute three major steps: ASM, Exadata Storage, and ASM again.

For ASM

Inside ASM we can use this query to collect some information about the diskgroups:

SQL> col name format a12 head 'Disk Group';
SQL> col total_mb format 999999999 head 'Total GB|Raw';
SQL> col free_mb format 999999999 head 'Free GB|Raw';
SQL> col avail_mb format 999999999 head 'Total GB|Usable';
SQL> col usable_mb format 999999999 head 'Free GB|Usable';
SQL> col usable_mb format 999999999 head 'Free GB|Usable';
SQL> col cdisks format 99999 head 'Cell|Disksl';
SQL>
SQL> select a.name,a.total_mb,a.free_mb,a.type,
  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,
  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,
  4  count(b.path) cdisks
  5  from v$asm_diskgroup a, v$asm_disk b
  6  where a.group_number=b.group_number
  7  group by a.name,a.total_mb,a.free_mb,a.type,
  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,
  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)
 10  order by 2,1
 11  /

               Total GB    Free GB          Total GB    Free GB   Cell
Disk Group          Raw        Raw TYPE       Usable     Usable Disksl
------------ ---------- ---------- ------ ---------- ---------- ------
RECOC3          4239360    2465540 NORMAL       2070       1204     60
DATAC3         15790080    2253048 NORMAL       7710       1100     60

SQL>
SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks
  2  from v$asm_disk d, v$asm_diskgroup dg
  3  where d.group_number = dg.group_number
  4  and dg.name IN ('DATAC3')
  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status
  6  order by 1,2,3;

NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS
------ ------------------------------ -------- ------------ ------- ------- ----------
DATAC3 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC3 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC3 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC3 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12
DATAC3 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12

SQL>

With that, I have three important information: number the disks (60), redundancy type (NORMAL), total actual size (RAW value – 15790080). To discover the size for each disk in ASM (here I do manually and not check in v$asm_disk just to show you the steps and to be more didact) you can divide the raw space/#disks:

SQL> SELECT (15790080/1024)/60 as gbDISK FROM dual;

    GBDISK
----------
       257

SQL>

So, each disk has 257GB of space size (in raw). Since the actual free space is 1100GB (1.07TB) and we want to add more 2TB we need to increase the value for each disk.

The formula is simple: NewValue(inGB)*#OfDisksPerCell*#NumberofCells. Here I choose 330GB per disk, so, the new size for diskgroup will be:

SQL> SELECT (330*12*5) AS newsizeGB FROM dual;

 NEWSIZEGB
----------
     19800

SQL>

But this value is not correct because does not consider the mirror type, so, we need to divide this value. If it NORMAL, divide by 2, if HIGH, divide by 3. To compare, the old and new expected value:

SQL> SELECT (257*12*5)/2 as actualsizeGBUsable, (330*12*5)/2 AS newsizeGBUSable FROM dual;

ACTUALSIZEGBUSABLE NEWSIZEGBUSABLE
------------------ ---------------
              7710            9900

SQL>

So, the new total space for diskgroup will be around 9.6 TB (9900 GB). And we will add (as free space) around 2.1 TB. Probably you need to execute these formulas more than one time to find the desired size per disk.

I start to calculate by disk (and after discovering the final diskgroup size) instead of starting with diskgroup size (and dividing to discover the disk size) because doing this way, the size for disk will be always correct and align with storage cell grid disk. Remember that grid disks are aligned in 16MB and, if you start to choose one arbitrary value to the max size for the ASM diskgroup, you can reach a value per grid disk that is not 16MB aligned. As an example, if I start choosing 20TB for diskgroup, the size per disk will be (20*1024)/60 = 341.33GB and this is not aligned with 16MB.

For 16 Mb explanation, you can check in the Exadata docs

Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.

For Storage Cell

In the Exadata side, first, check some info about the actual state for grid disks. Here I connect in one cell (if you want you can use dcli to call every/all cells) and check some info for the grid disk:

[root@exaceladm01 ~]# cellcli
CellCLI: Release 18.1.6.0.0 - Production on Fri Jun 21 16:57:49 CEST 2019

Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.

CellCLI> list celldisk
         CD_00_exaceladm01       normal
         CD_01_exaceladm01       normal
         CD_02_exaceladm01       normal
         CD_03_exaceladm01       normal
         CD_04_exaceladm01       normal
         CD_05_exaceladm01       normal
         CD_06_exaceladm01       normal
         CD_07_exaceladm01       normal
         CD_08_exaceladm01       normal
         CD_09_exaceladm01       normal
         CD_10_exaceladm01       normal
         CD_11_exaceladm01       normal
         FD_00_exaceladm01       normal
         FD_01_exaceladm01       normal
         FD_02_exaceladm01       normal
         FD_03_exaceladm01       normal

CellCLI> list celldisk CD_02_exaceladm01 detail
         name:                   CD_02_exaceladm01
         comment:
         creationTime:           2016-11-29T10:23:35+01:00
         deviceName:             /dev/sdc
         devicePartition:        /dev/sdc
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              2.6688079833984375T
         id:                     d57b31bb-6043-4cea-b992-ef8075f42e77
         physicalDisk:           PUT81V
         size:                   7.152252197265625T
         status:                 normal

CellCLI>

CellCLI> list griddisk where name like 'DATAC3.*';
         DATAC3_CD_00_exaceladm01        active
         DATAC3_CD_01_exaceladm01        active
         DATAC3_CD_02_exaceladm01        active
         DATAC3_CD_03_exaceladm01        active
         DATAC3_CD_04_exaceladm01        active
         DATAC3_CD_05_exaceladm01        active
         DATAC3_CD_06_exaceladm01        active
         DATAC3_CD_07_exaceladm01        active
         DATAC3_CD_08_exaceladm01        active
         DATAC3_CD_09_exaceladm01        active
         DATAC3_CD_10_exaceladm01        active
         DATAC3_CD_11_exaceladm01        active

CellCLI>
CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;
         name:                   DATAC3_CD_04_exaceladm01
         asmDiskGroupName:       DATAC3
         asmDiskName:            DATAC3_CD_04_EXACELADM01
         asmFailGroupName:       EXACELADM01
         availableTo:
         cachedBy:               FD_00_exaceladm01
         cachingPolicy:          default
         cellDisk:               CD_04_exaceladm01
         comment:                "Cluster exa-cl3 diskgroup DATAC3"
         creationTime:           2017-01-20T17:23:21+01:00
         diskType:               HardDisk
         errorCount:             0
         id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778
         size:                   257G
         status:                 active

CellCLI>

Here you can see that I checked:

  • The celldisks info for this cell.
  • Detail for one celldisk (look the freeSpace attribute to verify if you have free space).
  • The grid disks for this cell.
  • Details for the griddisk (look that the size is the same value that I calculated manually).

This part was just to check and show you how to verify some info, with time, you don’t need to check this in every maintenance (because you will be familiar with the environment). Be careful that, if you have different grid disk space division per storage cells, you need to check if you have available space in all your storage celldisks.

To expand the grid disks you have two options, enter in each cell and expand manually one by one, or create one script and call by dcli (the option that I choose). So, create one script that executes the ALTER GRIDDISK command for the new desired size. Just remember to be careful and choose the correct grid disks (here is for VM 03, that means DATAC3):

[DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh
dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM01 size=330G;
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM02 size=330G;
…
dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM02 size=330G;
dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM03 size=330G;
dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM04 size=330G;
dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM05 size=330G;
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh
exaceladm01: GridDisk DATAC3_CD_00_exaceladm01 successfully altered
exaceladm02: GridDisk DATAC3_CD_00_exaceladm02 successfully altered
exaceladm03: GridDisk DATAC3_CD_00_exaceladm03 successfully altered
…
…
exaceladm01: GridDisk DATAC3_CD_11_exaceladm01 successfully altered
exaceladm02: GridDisk DATAC3_CD_11_exaceladm02 successfully altered
exaceladm03: GridDisk DATAC3_CD_11_exaceladm03 successfully altered
exaceladm04: GridDisk DATAC3_CD_11_exaceladm04 successfully altered
exaceladm05: GridDisk DATAC3_CD_11_exaceladm05 successfully altered
[DOM0 - root@exadbadm01 tmp]$
[DOM0 - root@exadbadm01 tmp]$

Above I cropped the output to reduce the size or post, but you can check the raw output here. After the change you can check the info for the grid disk:

[root@exaceladm01 ~]# cellcli
CellCLI: Release 18.1.6.0.0 - Production on Fri Jun 21 17:24:03 CEST 2019

Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.

CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;
         name:                   DATAC3_CD_04_exaceladm01
         asmDiskGroupName:       DATAC3
         asmDiskName:            DATAC3_CD_04_EXACELADM01
         asmFailGroupName:       EXACELADM01
         availableTo:
         cachedBy:               FD_00_exaceladm01
         cachingPolicy:          default
         cellDisk:               CD_04_exaceladm01
         comment:                "Cluster exa-cl3 diskgroup DATAC3"
         creationTime:           2017-01-20T17:23:21+01:00
         diskType:               HardDisk
         errorCount:             0
         id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778
         size:                   330G
         status:                 active

CellCLI> list celldisk CD_02_exaceladm01 detail
         name:                   CD_02_exaceladm01
         comment:
         creationTime:           2016-11-29T10:23:35+01:00
         deviceName:             /dev/sdc
         devicePartition:        /dev/sdc
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              2.5975189208984375T
         id:                     d57b31bb-6043-4cea-b992-ef8075f42e77
         physicalDisk:           PUT81V
         size:                   7.152252197265625T
         status:                 normal

CellCLI> exit
quitting

[root@exaceladm01 ~]#

For ASM – Part #2

After you change the grid disks in the storage side, you can go back to ASM and extend the diskgroup:

SQL> ALTER DISKGROUP DATAC3 RESIZE ALL;

Diskgroup altered.

SQL>

And you can check that the size was already added (look that values hit what we calculated before):

SQL> select a.name,a.total_mb,a.free_mb,a.type,
  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,
  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,
  4  count(b.path) cdisks
  5  from v$asm_diskgroup a, v$asm_disk b
  6  where a.group_number=b.group_number
  7  group by a.name,a.total_mb,a.free_mb,a.type,
  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,
  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)
 10  order by 2,1
 11  /

               Total GB    Free GB          Total GB    Free GB   Cell
Disk Group          Raw        Raw TYPE       Usable     Usable Disksl
------------ ---------- ---------- ------ ---------- ---------- ------
RECOC3          4239360    2465540 NORMAL       2070       1204     60
DATAC3         20275200    6738128 NORMAL       9900       3290     60

SQL>

And you can check the v$asm_operation to check the rebalance progress:

SQL> select operation, EST_MINUTES, EST_RATE, EST_WORK, sofar from v$asm_operation;

OPERA EST_MINUTES   EST_RATE   EST_WORK      SOFAR
----- ----------- ---------- ---------- ----------
REBAL           0          0          0          0
REBAL           0      46158      18490       5761
REBAL           0          0          0          0
REBAL           0          0          0          0
REBAL           0          0          0          0

SQL>

Conclusion

As you can see, the steps to do that are simple and not complex, you just need to take care about some details of your environment: Number of the disks per cell, number the cells and the VM where you want to add the space is critical to do the correct change. Remember to align with 16MB the size of your grid disk, when you are adding it is not a big deal, but if you want to shrink this can break your ASM diskgroup.

Check that the only change effectively is the size of the grid disk, all the others occur automatically because of the grid disk. ASM diskgroup will increase to the max value that it is available and space is available just after the command.

The steps above are more detailed that you will do in daily maintenance, but help you to understand most of the datils for this kind of change.

 

Reference:

How to Resize Grid Disks in Exadata (Doc ID 2176737.1) – https://support.oracle.com/epmos/faces/DocContentDisplay?id=2176737.1

Resizing Grid Disks – https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-570A0C37-907C-4417-BC93-AC4ABAF7E3AD 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

3 thoughts on “Increase size for Exadata Grid Disks

  1. Pingback: Shrink ASM Diskgroup and Exadata Grid Disks | Blog Fernando Simon

  2. Jignesh

    YOu are confusing…

    SQL> SELECT (15790080/1024)/60 as gbDISK FROM dual;

    GBDISK
    ———-
    257

    SQL>

    In above , 15790080 is in GB …..only….then why are you dividing by 1024 again..??

    Reply
    1. Simon Post author

      Hi,
      The error actually is related to the output of the volume of the select. If you check, the column is “free_mb”, but the format was made wrongly and shows as “GB”instead of raw “MB”.
      So, you still need to divide by 1024.
      Thanks for reading and pointing this issue.
      Best regards.

      Fernando Simon

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *