Exadata and ZDLRA, Disable HAIP

HAIP (High Availability IP) is not supported for the Exadata environment but can occur (if you did not create the cluster using OEDA) that HAIP became in use. And this particularity true for ZDLRA. So, during the upgrade from the previous version (12.2) to a higher version, it is needed to remove HAIP.

Usually, when we upgrading from 12.2 to 18c the HAIP is removed from Exadata. If the upgrade is from 12.1, and HAIP is there, it continues and is not removed by the upgrade process. If you are using HAIP and your GI is 12.1, this procedure as-is described here can’t be used (need some adaptation), because of some requirements from ASM+ACFS+DB. But since this is a preliminary step from a GI upgrade, the focus is to disable and remove it from GI.

The HAIP is not needed for Exadata because by architecture the InfiniBand network already defines (per server) two IP’s to avoid the single point of failure. So, it is not needed to create an additional layer (HAIP and virtual IP), that does the same that already exists by network design.

*Image removed from Oracle Presentation: Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour Oct 2019

This procedure can be used for a non-Exadata environment too. But before continue it is important to be aware of some details with ACFS. If you use ACFS, be careful since the HAIP IP is used in the ASM proxy and needs to be updated; and if ACFS is in place, this procedure is valid just for GI 12.2+.

Environment

In this scenario, I am removing HAIP from ZDLRA GI. The HAIP has configured automatically and we are using ACFS.

The focus is to disable HAIP and remove it from CRS. Another detail is that we need to remove the dependencies between the CRS resources (like ASM that need HAIP) too.

One important detail is that since we are removing the HAIP, we need to swap the HAIP IP that it is used by services (like ASM Proxy) for something that exists. So, you need to have high availability IP’s to use. For Exadata, we use the interconnect IP’s, but if you are using in another environment, be careful that your network has the needed requirements (throughput, failover, and others).

GI Upgrade

During the GI upgrade from 12.2 to 19c we need to run the runcluvfy and it will detect if HAIP is enabled:

....
....
Checks did not pass for the following nodes:
        zeroserv02,zeroserv01


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Node Connectivity ...FAILED
zeroserv02: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes
            "zeroserv01,zeroserv02".

zeroserv01: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes
            "zeroserv01,zeroserv02".

Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.


CVU operation performed:      stage -pre crsinst
Date:                         Dec 9, 2019 1:38:46 PM
CVU home:                     /u01/app/19.0.0.0/grid/
User:                         oracle
[root@zeroserv01 ~]#

Remove HAIP

The steps need to be executed in order to avoid errors during the procedure. Unfortunately, it is needed to have one maintenance window due to the unavailability of the services. The running databases need to be shutdown due to the CRS restart by an example.

Bellow, the steps are summarized and explained to be followed. Please be aware of the IP’s involved (they will be different in your case).

1 – Shutdown services

So, this first point is to shutdown all databases running in this cluster:

[oracle@zeroserv01 ~]$ srvctl stop database -d zdlras -o immediate
[oracle@zeroserv01 ~]$

2- Check HAIP, ASM Proxy, and ACFS

It is needed to check if HAIP is enabled. Another detail is check for ASM proxy, and ACFS too:

[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.cluster_interconnect.haip -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       zeroserv01               STABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.proxy_advm -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.proxy_advm
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res -w "TYPE = ora.acfs.type" -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.catalog.raadmin.acfs
               ONLINE  ONLINE       zeroserv01               mounted on /raacfs/r
                                                             aadmin,STABLE
               ONLINE  ONLINE       zeroserv02               mounted on /raacfs/r
                                                             aadmin,STABLE
ora.catalog.raosbadmin.acfs
               ONLINE  ONLINE       zeroserv01               mounted on /osbcat,S
                                                             TABLE
               ONLINE  ONLINE       zeroserv02               mounted on /osbcat,S
                                                             TABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$

3 – Check New IP

The next step is to check the new IP that will be used. Since ASM does not use HAIP, we can (at Exadata) pickup the same used IP:

[oracle@zeroserv01 ~]$ export ORACLE_SID=+ASM1
[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Mon Dec 9 15:00:55 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string      192.168.10.1:192.168.10.2
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/oifcfg getif
bondeth0  10.208.68.0  global  public
ib0  192.168.8.0  global  cluster_interconnect,asm
ib1  192.168.8.0  global  cluster_interconnect,asm
[oracle@zeroserv01 ~]$

In this case, the IP that will substitute HAIP IP is 192.168.10.1/192.168.10.2. And, as you can see above, they are from the interconnect network.

4 – Fixing IP for ASM Proxy

The next step is to fix the IP that is used by the ASM Proxy instance. But default, it pickup the HAIP IP during the startup. The idea here is to force the same IP than the ASM instance for cluster_interconnect parameter.

Connecting at APX instance

Connect at APX instance, check the parameters (and see that are without values). After that we create one init file from memory to have the backup:

[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv01 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:20:59 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>  show parameter instance_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      +APX1
SQL>  show parameter instance_type

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_type                        string      ASMPROXY
SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string
SQL> create pfile = '/tmp/pfileapx1' from memory;

File created.

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$

Changing the IP

For ASM Proxy instance we need to set the IP for cluster_interconnect parameter with the same IP used by ASM instance.

To do that, we need to set it in init for ASM Proxy. But since the ASM Proxy instance doesn’t have one, we create one at $GI_HOME/dbs with the connect name (look below the folder and instance name). Below I set just the cluster to interconnect parameter and dedicated to the instance running in this node.

[oracle@zeroserv01 ~]$ echo "+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'" > $ORACLE_HOME/dbs/init+APX1.ora
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ cat $ORACLE_HOME/dbs/init+APX1.ora
+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'
[oracle@zeroserv01 ~]$

Restart the ASM Proxy

After creating the init, we restart the ASM Proxy in this node.

[root@zeroserv01 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid
[root@zeroserv01 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv01 -force
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv01
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# ps -ef |grep APX
oracle   267811      1  0 10:43 ?        00:00:00 apx_pmon_+APX1
oracle   267813      1  0 10:43 ?        00:00:00 apx_clmn_+APX1
oracle   267815      1  0 10:43 ?        00:00:00 apx_psp0_+APX1
oracle   267820      1  1 10:43 ?        00:00:00 apx_vktm_+APX1
oracle   267824      1  0 10:43 ?        00:00:00 apx_gen0_+APX1
oracle   267826      1  0 10:43 ?        00:00:00 apx_mman_+APX1
oracle   267830      1  0 10:43 ?        00:00:00 apx_gen1_+APX1
oracle   267834      1  0 10:43 ?        00:00:00 apx_diag_+APX1
oracle   267836      1  0 10:43 ?        00:00:00 apx_dskm_+APX1
oracle   267838      1  0 10:43 ?        00:00:00 apx_pman_+APX1
oracle   267840      1  0 10:43 ?        00:00:00 apx_dia0_+APX1
oracle   267842      1  0 10:43 ?        00:00:00 apx_lreg_+APX1
oracle   267845      1  0 10:43 ?        00:00:00 apx_pxmn_+APX1
oracle   267847      1  0 10:43 ?        00:00:00 apx_rbal_+APX1
oracle   267849      1  0 10:43 ?        00:00:00 apx_vbg0_+APX1
oracle   267851      1  0 10:43 ?        00:00:00 apx_vdbg_+APX1
oracle   267853      1  0 10:43 ?        00:00:00 apx_vubg_+APX1
root     267979  32720  0 10:43 pts/0    00:00:00 grep --color=auto APX
[root@zeroserv01 ~]#

Look that instance restarted correctly and it is up.

And we can check the parameter if it is OK with the IP:

[oracle@zeroserv01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:51:59 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string      192.168.10.1:192.168.10.2
SQL> show parameter instance_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      +APX1
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$

If the instance not startup (or the IP’s are wrong) you need to check the init file that you created in $GIHOME/dbs folder.

Restart ACFS

Since we restarted the ASM Proxy instance, the ACFS mountpoints went down and need to be up again:

[root@zeroserv01 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv01
CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'
CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'
CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'
CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv01'
CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded
CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded
[root@zeroserv01 ~]#

I started up using the CRS resource for that.

Other Nodes

After it is Ok for node 1, we can do it for other nodes. Here, since I made for ZDLRA, I have just the second node to fix ASM Proxy. Be careful with the IP’s used, they are specific for this node and remember to use the correct init file.

[root@zeroserv02 ~]# su - oracle
Last login: Tue Dec 10 10:53:02 CET 2019
[oracle@zeroserv02 ~]$ export ORACLE_SID=+ASM2
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:56:24 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string      192.168.10.3:192.168.10.4
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:57:15 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>
SQL> show parameter instance_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      +APX2
SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string
SQL> create pfile = '/tmp/pfileapx2' from memory;

File created.

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ ls -l $ORACLE_HOME/dbs/init?APX*
ls: cannot access /u01/app/12.2.0.1/grid/dbs/init?APX*: No such file or directory
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ echo "+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'" > $ORACLE_HOME/dbs/init+APX2.ora
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ cat $ORACLE_HOME/dbs/init+APX2.ora
+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ exit    
[root@zeroserv02 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid
[root@zeroserv02 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv02 -force
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv02
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv02
CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'
CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'
CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv02'
CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# su - oracle
Last login: Tue Dec 10 10:54:10 CET 2019
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2

[oracle@zeroserv02 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 11:00:46 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter cluster_interconnects

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects                string      192.168.10.3:192.168.10.4
SQL> show parameter instance_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      +APX2
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
You have new mail in /var/spool/mail/oracle
[oracle@zeroserv02 ~]$

5 – ASM Dependency

Since the ASM since the CRS depends on HAIP we need to remove this dependency. Here we have a tricky part. It is needed to completely change the dependencies for ASM, setting it to CRS/CSS directly.

First, at first node we check the current dependency for START and STOP:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep START_DEPENDENCIES=
START_DEPENDENCIES=hard(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) weak(ora.LISTENER.lsnr) pullup(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) dispersion:active(site:type:ora.asm.type)
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep STOP_DEPENDENCIES=
STOP_DEPENDENCIES=hard(intermediate:ora.ASMNET1LSNR_ASM.lsnr,intermediate:ora.ASMNET2LSNR_ASM.lsnr)
[root@zeroserv01 ~]#

And now we change it. Look at the parameters values. They are completely different, and need to be with these specific values:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@zeroserv01 ~]#

If you see closely the HAIP it is not listed there as a dependency, setting with the values, we completely remove HAIP dependency and inheritance.

And we need to do the same for the other node:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@zeroserv02 ~]#

6 – Disable HAIP Resource

The next step is to disable the HAIP resource from startup at CRS. We do this in both nodes.

Node 1:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@zeroserv01 ~]#

Node 2:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@zeroserv02 ~]#

But check that we do not stop yet the HAIP resource. It is needed to remain up at this moment.

If we check with the init for CRS, it is still there in node 1:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.crf
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.crsd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.cssd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.ctssd
      1        ONLINE  ONLINE       zeroserv01               OBSERVER,STABLE
ora.diskmon
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.drivers.oka
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.gipcd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.gpnpd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.mdnsd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.storage
      1        ONLINE  ONLINE       zeroserv01               STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#

And for the other node too:

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.crf
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.crsd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cssd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.ctssd
      1        ONLINE  ONLINE       zeroserv02               ACTIVE:0,STABLE
ora.diskmon
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.drivers.oka
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.gipcd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.gpnpd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.mdnsd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.storage
      1        ONLINE  ONLINE       zeroserv02               STABLE
--------------------------------------------------------------------------------
[root@zeroserv02 ~]#

7 – Restart CRS

After we configure the dependencies for ASM we can restart CRS to shutdown HAIP (and test the changes that we made). I prefer to execute, first, in just one node and if everything goes well, I do for the others.

Stop Cluster

So, first the STOP CLUSTER in the first node:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f
CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv01'
CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.cvu' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'zeroserv01'
CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'
CRS-2677: Stop of 'ora.cvu' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.zeroserv01.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.scan3.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.zeroserv01.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv01' succeeded
CRS-2675: Stop of 'ora.chad' on 'zeroserv02' failed
CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv02'
CRS-2675: Stop of 'ora.chad' on 'zeroserv01' failed
CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv01'
CRS-2681: Clean of 'ora.chad' on 'zeroserv02' succeeded
CRS-2681: Clean of 'ora.chad' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'zeroserv01'
CRS-2677: Stop of 'ora.mgmtdb' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'zeroserv01'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.qosmserver' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.cvu' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.zeroserv01.vip' on 'zeroserv02'
CRS-2676: Start of 'ora.cvu' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.zeroserv01.vip' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02'
CRS-2676: Start of 'ora.scan3.vip' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02'
CRS-2676: Start of 'ora.MGMTLSNR' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'zeroserv02'
CRS-2676: Start of 'ora.qosmserver' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'zeroserv02'
CRS-2676: Start of 'ora.chad' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv01'
CRS-2677: Stop of 'ora.ons' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv01'
CRS-2677: Stop of 'ora.net1.network' on 'zeroserv01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv01' has completed
CRS-2677: Stop of 'ora.crsd' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv01'
CRS-2677: Stop of 'ora.storage' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'
CRS-2677: Stop of 'ora.ctssd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv01'
CRS-2677: Stop of 'ora.cssd' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv01'
CRS-2677: Stop of 'ora.diskmon' on 'zeroserv01' succeeded
[root@zeroserv01 ~]#

Stop/Start CRS

And if everything was successful, we can stop CRS. This is needed because HAIP is from init from CRS:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.crf' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv01'
CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'zeroserv01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@zeroserv01 ~]#

After that we can start again CRS in this node:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@zeroserv01 ~]#

Check CRS Init

And after some time we can see if the  HAIP not restarted (as expected):

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       zeroserv01               Started,STABLE
ora.cluster_interconnect.haip
      1        OFFLINE OFFLINE                               STABLE
ora.crf
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.crsd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.cssd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.ctssd
      1        ONLINE  ONLINE       zeroserv01               ACTIVE:0,STABLE
ora.diskmon
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.drivers.oka
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.gipcd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.gpnpd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.mdnsd
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.storage
      1        ONLINE  ONLINE       zeroserv01               STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#

As you can see above, the HAIP not started during the init of CRS. If the HAIP still up, please check the topic 5.

Other nodes

After that, we can do the same in the other nodes.

[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f
CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv02'
CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02'
CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.zeroserv02.vip' on 'zeroserv02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'zeroserv02'
CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02'
CRS-2677: Stop of 'ora.zeroserv02.vip' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.chad' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'zeroserv01'
CRS-2672: Attempting to start 'ora.zeroserv02.vip' on 'zeroserv01'
CRS-2676: Start of 'ora.zeroserv02.vip' on 'zeroserv01' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv02'
CRS-2677: Stop of 'ora.ons' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv02'
CRS-2677: Stop of 'ora.net1.network' on 'zeroserv02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv02' has completed
CRS-2677: Stop of 'ora.crsd' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv02'
CRS-2677: Stop of 'ora.storage' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'
CRS-2677: Stop of 'ora.ctssd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv02'
CRS-2677: Stop of 'ora.cssd' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv02'
CRS-2677: Stop of 'ora.diskmon' on 'zeroserv02' succeeded
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.crf' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv02'
CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'zeroserv02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@zeroserv02 ~]# 
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cluster_interconnect.haip
      1        OFFLINE OFFLINE                               STABLE
ora.crf
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.crsd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cssd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.ctssd
      1        ONLINE  ONLINE       zeroserv02               ACTIVE:0,STABLE
ora.diskmon
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.drivers.oka
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.gipcd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.gpnpd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.mdnsd
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.storage
      1        ONLINE  ONLINE       zeroserv02               STABLE
--------------------------------------------------------------------------------
[root@zeroserv02 ~]#

8 – Check if everything is UP

After doing the changes in both nodes we can see if everything is up and running:

[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ob_dbfs
               OFFLINE OFFLINE      zeroserv01               STABLE
               OFFLINE OFFLINE      zeroserv02               STABLE
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.ASMNET2LSNR_ASM.lsnr
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.CATALOG.RAADMIN.advm
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.CATALOG.RAOSBADMIN.advm
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.CATALOG.dg
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.DELTA.dg
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.catalog.raadmin.acfs
               ONLINE  ONLINE       zeroserv01               mounted on /raacfs/r
                                                             aadmin,STABLE
               ONLINE  ONLINE       zeroserv02               mounted on /raacfs/r
                                                             aadmin,STABLE
ora.catalog.raosbadmin.acfs
               ONLINE  ONLINE       zeroserv01               mounted on /osbcat,S
                                                             TABLE
               ONLINE  ONLINE       zeroserv02               mounted on /osbcat,S
                                                             TABLE
ora.chad
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.net1.network
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.ons
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
ora.proxy_advm
               ONLINE  ONLINE       zeroserv01               STABLE
               ONLINE  ONLINE       zeroserv02               STABLE
rep_dbfs
               OFFLINE OFFLINE      zeroserv01               STABLE
               OFFLINE OFFLINE      zeroserv02               STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       zeroserv01               192.168.10.1 192.168
                                                             .10.2,STABLE
ora.asm
      1        ONLINE  ONLINE       zeroserv01               Started,STABLE
      2        ONLINE  ONLINE       zeroserv02               Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       zeroserv01               Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       zeroserv02               STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.zdlras.db
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.zeroserv01.vip
      1        ONLINE  ONLINE       zeroserv01               STABLE
ora.zeroserv02.vip
      1        ONLINE  ONLINE       zeroserv02               STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#

As you can see, everything is up. ASM is online in both nodes, as the listeners for ASM too.

9 – Start Databases

To finish, we need to restart all databases that were stopped at topic 1.

[oracle@zeroserv01 ~]$ srvctl start database -d zdlras
[oracle@zeroserv01 ~]$

Clean up

After you complete the GI upgrade for 19c, you can remove the ASM Proxy init file (from $GI_HOME/dbs) that was created before. It is not needed anymore because the upgrade removes completely the HAIP from CRS because it is not in use.

Conclusion

As told before, HAIP is not supported by the Exadata environment. If you check the definition for HAIP and the hardware design for the Exadata/Engineering System, it is completely redundant. The InfiniBand network and the dual-channel/ports have the same effect (even better)

But unfortunately, HAIP can be up for several reasons (like ZDLRA) and because of the way that ASM Proxy starts and pick up the interconnect IP, HAIP can be selected. Remove HAIP it is a little tricky because of the ASM dependency, we need to set some specific parameters.

As you saw before, there are several steps. But you can use and adapt them to your environment. If it is Exadata/Engineering System (like ZDLRA) they can be used almost as-is (just check the IPs). If it is not, read and adapt.

References:

 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

2 thoughts on “Exadata and ZDLRA, Disable HAIP

  1. Pingback: ZDLRA, Patch the Recovery Appliance | Fernando Simon

Leave a Reply

Your email address will not be published. Required fields are marked *