Is common that our systems grow with time, and the environment that sustains it needs to improve. And the same occurs for ZDLRA. Imagine that now you added a new datacenter and bought a new ZDLRA and want to replicate between them, or that now you want to enable the replication, configuring it.
This is possible and is not complicated to do, and I will show here how to do that. So, in this post, I will show how to configure the replication network for ZDLRA that was already deployed. Basically a post-install procedure.
Replication network
In my previous post, I already explain the basics of how the replication works for ZDLRA. And going deep about the replication, the ZDLRA have a physical dedicated network to replicate:
And according to documentation:
“The optional replication network connects the local Recovery Appliance (the upstream appliance) with a remote Recovery Appliance (the downstream appliance). Oracle recommends a broadband, encrypted network, instead of an insecure public network, wherever possible.”
It is optional because you can share the same “Ingest Network” to receive replication, but is not recommended. But whatever the network mode that you choose the “Replication Network” is a different subnet that not overlaps/is part of the “Ingest Network”.
ZDLRA Environment
In this post I have two ZDLRA’s to enable the replication. But this can vary in your scenario. So, I will show how to do in both nodes, in both ZDLRA’s (will be 4 to do).
So, use this as a guide on how to do, but remember to adapt to your environment. Ip’s, physical networks (bond or no), and routes are an example of details that will change. I will focus on how to configure the GI and ZDLRA in this post.
And the most important, I recommend opening a proactive SR at MOS informing what are you doing and asking doubts before the start. A good start point is a note “Post Install – Replication Network Configuration for ZDLRA (Doc ID 2126047.1)” at MOS.
Basic network config
The network configuration follows the same as normal Linux network configuration: you need to define the separated ip’s for both ZDLRA’s, if you will use bond, or the route table. Since this depends on every environment I will not cover here.
But basically:
- Hostname: You need to choose at least: 1 Hostname at your replication network for each node, 1 Hostname for VIP for each node
- Scan Hostname: 1 Hostname to be used as a scan for replication data exchange. The scan will be used for each ZDLRA that you are configuring.
- Configure network files: Configure the ifcfg files. The files ifcfg-eth2, ifcfg-eth3 and ifcfg-bondeth1 if you will use bond. If just ifcfg-eth2 if you will use a single connection. This is done in all nodes of the ZDLRA appliance.
- Route configuration: You need to guarantee that access made by the replication network is not routed through the normal ingest network. This depends on the way that you have your network architecture, but maybe you need to change the files “route-*” in each node.
My current system is:
- ZDLRAS1: ZDLRA installed on site 1, have two nodes: zdlras1n1 and zdlras1n2. It will be upstream (who send backups).
- ZDLRAS2: ZDLRA installed on site 2, have two nodes: zdlras2n1 and zdlras2n2. It will be downstream (who received backup).
What I will add:
- ZDLRAS1: I will add: zdlras1n1-rvip (200.254.255.21) as replication vip for zdlras1n1, zdlras1n2-rvip (200.254.255.22) as replication vip for zdlras1n2. Scan zdlras1-rep.
- ZDLRAS2: I will add: zdlras1n1-rvip (200.254.255.23) as replication vip for zdlras2n1, zdlras2n2-rvip (200.254.255.24) as replication vip for zdlras2n2. Scan zdlras2-rep.
GI Configuration
The next step after the basic configuration is to configure the GI to add the network, vip, and scan.
Checking interfaces
After you configure the Linux part and the network basic configuration the GI can be check if the new interface can be used by GI. So, the first step is to check this in both nodes in both ZDLRA’s with command “oifcfg iflist” (I cropped below to show just what is needed):
############## Upstream ZDLRA [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/oifcfg iflist -p -n … bondeth1 200.254.255.0 UNKNOWN 255.255.255.0 … [root@zdlras1n1 ~]# ############## Downstream ZDLRA [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/oifcfg iflist -p -n … bondeth1 200.254.255.0 UNKNOWN 255.255.255.0 … [root@zdlras2n1 ~]#
Add network
Since the interfaces are visible and not used, we can add the network number 2 at GI level. This is done in just one node per ZDLRA. TO do that we use the command “srvctl add network” as root user in both ZDLRA’s that we are configuring:
############## Upstream ZDLRA [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl config network -k 2 PRCR-1001 : Resource ora.net2.network does not exist [root@zdlras1n1 ~]# [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add network -netnum 2 -subnet 200.254.255.0/255.255.255.0/bondeth1 [root@zdlras1n1 ~]# [root@zdlras1n1 ~]# srvctl config network -k 2 Network 2 exists Subnet IPv4: 200.254.255.0/255.255.255.0/bondeth1, static Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes: [root@zdlras1n1 ~]# ############## Downstream ZDLRA [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl config network -k 2 PRCR-1001 : Resource ora.net2.network does not exist [root@zdlras2n1 ~]# [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add network -netnum 2 -subnet 200.254.255.0/255.255.255.0/bondeth1 [root@zdlras2n1 ~]# [root@zdlras2n1 ~]# srvctl config network -k 2 Network 2 exists Subnet IPv4: 200.254.255.0/255.255.255.0/bondeth1, static Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes: [root@zdlras2n1 ~]#
Look above that we are using the bondeth1 as the interface for this network (the bondeth0 if for ingest network). And I tested if the network is available or no (need to be unused).
Add VIP
After we can add the VIP for each node for each ZDLRA that we are configuring (done using “srvctl add vip”):
############## Upstream ZDLRA [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras1n1 -A zdlras1n1-rvip.oralocal/255.255.255.0/bondeth1 -k 2 [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras1n2 -A zdlras1n2-rvip.oralocal/255.255.255.0/bondeth1 -k 2 [root@zdlras1n1 ~]# ############## Downstream ZDLRA [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras2n1 -A zdlras2n1-rvip.oralocal/255.255.255.0/bondeth1 -k 2 [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras2n2 -A zdlras2n2-rvip.oralocal/255.255.255.0/bondeth1 -k 2 [root@zdlras2n1 ~]#
Check that I defined the hostname for each vip for each node and the parameter “k” that defined the network where this vip will be created. Be careful with the “n” parameter that defined the node name.
Add Scan
After add vip we can add scan for each ZDLRA cluster with “srvctl add scan”:
############## Upstream ZDLRA [root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add scan -netnum 2 -scanname zdlras1-rep.oralocal [root@zdlras1n1 ~]# ############## Second ZDLRA [root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add scan -netnum 2 -scanname zdlras2-rep.oralocal [root@zdlras2n1 ~]#
Be careful to use the connect “netnum” parameter value (needs to point to the second network) and scan name.
Start Vip
After the scan we can start the vips for each node in each ZDLRA:
############## Upstream ZDLRA [root@zdlras1n1 ~]# srvctl status vip -i zdlras1n1-rvip.oralocal VIP 200.254.255.21 is enabled VIP 200.254.255.21 is running on node: zdlras1n1 [root@zdlras1n1 ~]# [root@zdlras1n1 ~]# srvctl status vip -i zdlras1n2-rvip.oralocal VIP 200.254.255.22 is enabled VIP 200.254.255.22 is running on node: zdlras1n2 [root@zdlras1n1 ~]# ############## Downstream ZDLRA [root@zdlras2n1 ~]# srvctl status vip -i zdlras2n1-rvip.oralocal VIP 200.254.255.23 is enabled VIP 200.254.255.23 is running on node: zdlras2n1 [root@zdlras2n1 ~]# [root@zdlras2n1 ~]# srvctl status vip -i zdlras2n2-rvip.oralocal VIP 200.254.255.24 is enabled VIP 200.254.255.24 is running on node: zdlras2n2 [root@zdlras2n1 ~]#
Create Listener
The last step for GI configuration is to create and start “listener” and “scan listener” in each ZDLRA cluster:
############## Upstream ZDLRA [oracle@zdlras1n1 ~]$ export ORACLE_HOME=/u01/app/19.0.0.0/grid [oracle@zdlras1n1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH [oracle@zdlras1n1 ~]$ [oracle@zdlras1n1 ~]$ srvctl add listener -l LISTENER_REPL -p 1522 -k 2 [oracle@zdlras1n1 ~]$ srvctl start listener -l LISTENER_REPL [oracle@zdlras1n1 ~]$ srvctl status listener -l LISTENER_REPL Listener LISTENER_REPL is enabled Listener LISTENER_REPL is running on node(s): zdlras1n1,zdlras1n2 [oracle@zdlras1n1 ~]$ [oracle@zdlras1n1 ~]$ srvctl add scan_listener -netnum 2 -listener LISTENER_REPL -endpoints TCP:1522 [oracle@zdlras1n1 ~]$ srvctl start scan_listener -netnum 2 [oracle@zdlras1n1 ~]$ srvctl status scan_listener -netnum 2 SCAN Listener LISTENER_REPL_SCAN1_NET2 is enabled SCAN listener LISTENER_REPL_SCAN1_NET2 is running on node zdlras1n1 SCAN Listener LISTENER_REPL_SCAN2_NET2 is enabled SCAN listener LISTENER_REPL_SCAN2_NET2 is running on node zdlras1n2 SCAN Listener LISTENER_REPL_SCAN3_NET2 is enabled SCAN listener LISTENER_REPL_SCAN3_NET2 is running on node zdlras1n1 [oracle@zdlras1n1 ~]$ ############## Downstream ZDLRA [root@zdlras2n1 ~]# su - oracle Last login: Sat Nov 23 21:28:03 CET 2019 [oracle@zdlras2n1 ~]$ export ORACLE_HOME=/u01/app/19.0.0.0/grid [oracle@zdlras2n1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH [oracle@zdlras2n1 ~]$ [oracle@zdlras2n1 ~]$ srvctl add listener -l LISTENER_REPL -p 1522 -k 2 [oracle@zdlras2n1 ~]$ srvctl start listener -l LISTENER_REPL [oracle@zdlras2n1 ~]$ srvctl status listener -l LISTENER_REPL Listener LISTENER_REPL is enabled Listener LISTENER_REPL is running on node(s): zdlras2n1,zdlras2n2 [oracle@zdlras2n1 ~]$ [oracle@zdlras2n1 ~]$ [oracle@zdlras2n1 ~]$ srvctl add scan_listener -netnum 2 -listener LISTENER_REPL -endpoints TCP:1522 [oracle@zdlras2n1 ~]$ srvctl start scan_listener -netnum 2 [oracle@zdlras2n1 ~]$ srvctl status scan_listener -netnum 2 SCAN Listener LISTENER_REPL_SCAN1_NET2 is enabled SCAN listener LISTENER_REPL_SCAN1_NET2 is running on node zdlras2n1 SCAN Listener LISTENER_REPL_SCAN2_NET2 is enabled SCAN listener LISTENER_REPL_SCAN2_NET2 is running on node zdlras2n2 SCAN Listener LISTENER_REPL_SCAN3_NET2 is enabled SCAN listener LISTENER_REPL_SCAN3_NET2 is running on node zdlras2n1 [oracle@zdlras2n1 ~]$ [oracle@zdlras2n1 ~]$
The important detail here:
- Port: it is 1522. Look at the “endpoint” parameter.
- Network: Check that listener is added ate network #2 (parameter “k”).
- Listener name: we use LISTENER_REPL to follow the ZDLRA default config.
- User: All the commands are run with oracle
All the commands are executed in just one node of the cluster because they affect all nodes at the same time.
With that now with have the network configured in each node of both ZDLRA’s, all the vips and scan up and running, and also the listening to this new network.
ZDLRA Configuration
Since we have all nodes with everything configured, we can add this network ad ZDLRA config. The idea for ZDLRA is to allow it to receive and send backups trough this new network, and this is incredibly easy.
We need to change just two parameters at ZDLRA config tables: REPLICATION_IP_ADDRESS and BACKUP_IP_ADDRESS. The configuration change internal ZDLRA tables, so, before doing that review and check with Oracle Support at MOS if everything is OK and you can proceed.
REPLICATION_IP_ADDRESS
This configuration resides at the internal “rasys.host” table for ZDLRA. So, we update the table column:
############## Upstream ZDLRA [oracle@zdlras1n1 ~]$ sqlplus rasys/change^Me2 SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 23 23:22:09 2019 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Sat Nov 23 2019 21:33:26 +01:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> set linesize 250 SQL> SQL> col node_name format a50 SQL> col REPLICATION_IP_ADDRESS format a50 SQL> select node_name,replication_ip_address from host; NODE_NAME REPLICATION_IP_ADDRESS -------------------------------------------------- -------------------------------------------------- zdlras1n1.oralocal zdlras1n2.oralocal SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.21' where NODE_NAME = 'zdlras1n1.oralocal'; 1 row updated. SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.22' where NODE_NAME = 'zdlras1n2.oralocal'; 1 row updated. SQL> select node_name,replication_ip_address from host; NODE_NAME REPLICATION_IP_ADDRESS -------------------------------------------------- -------------------------------------------------- zdlras1n1.oralocal 200.254.255.21 zdlras1n2.oralocal 200.254.255.22 SQL> ############## Downstream ZDLRA [oracle@zdlras2n1 ~]$ sqlplus rasys/change^Me2 SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 23 23:39:13 2019 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Sat Nov 23 2019 19:54:17 +01:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> set linesize 250 SQL> col node_name format a50 SQL> col REPLICATION_IP_ADDRESS format a50 SQL> SQL> select node_name,replication_ip_address from host; NODE_NAME REPLICATION_IP_ADDRESS -------------------------------------------------- -------------------------------------------------- zdlras2n2.oralocal zdlras2n1.oralocal SQL> SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.23' where NODE_NAME = 'zdlras2n1.oralocal'; 1 row updated. SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.24' where NODE_NAME = 'zdlras2n2.oralocal'; 1 row updated. SQL> SQL> select node_name,replication_ip_address from host; NODE_NAME REPLICATION_IP_ADDRESS -------------------------------------------------- -------------------------------------------------- zdlras2n2.oralocal 200.254.255.24 zdlras2n1.oralocal 200.254.255.23 SQL>
As you can see above, the REPLICATION_IP_ADDRESS for each node reflects the IP at the replication network.
BACKUP_IP_ADDRESS
This configuration defines with IP’s the ZDLRA allows ingesting backups. Since the replication basically is a ingest coming from other network, we need to allow it. To do that we change the table internal table rai_host. And, of course, we can commit if everything is OK.
############## Upstream ZDLRA SQL> col ADMIN_IP_ADDRESS format a30 SQL> col BACKUP_IP_ADDRESS format a30 SQL> SELECT * FROM rai_host; NODE_NAME ADMIN_IP_ADDRESS BACKUP_IP_ADDRESS REPLICATION_IP_ADDRESS -------------------------------------------------- ------------------------------ ------------------------------ -------------------------------------------------- zdlras1n1.oralocal 10.160.5.1 10.160.5.2 200.254.255.21 zdlras1n2.oralocal 10.160.5.3 10.160.5.4 200.254.255.22 SQL> UPDATE rai_host SET backup_ip_address='200.254.255.21,'|| backup_ip_address WHERE node_name = 'zdlras1n1.oralocal'; 1 row updated. SQL> UPDATE rai_host SET backup_ip_address='200.254.255.22,'|| backup_ip_address WHERE node_name = 'zdlras1n2.oralocal'; 1 row updated. SQL> SELECT * FROM rai_host; NODE_NAME ADMIN_IP_ADDRESS BACKUP_IP_ADDRESS REPLICATION_IP_ADDRESS -------------------------------------------------- ------------------------------ ------------------------------ -------------------------------------------------- zdlras1n1.oralocal 10.160.5.1 200.254.255.21,10.160.5.2 200.254.255.21 zdlras1n2.oralocal 10.160.5.3 200.254.255.22,10.160.5.4 200.254.255.22 SQL> commit; Commit complete. SQL> ############## Downstream ZDLRA SQL> col ADMIN_IP_ADDRESS format a30 SQL> col BACKUP_IP_ADDRESS format a30 SQL> SQL> SELECT * FROM rai_host; NODE_NAME ADMIN_IP_ADDRESS BACKUP_IP_ADDRESS REPLICATION_IP_ADDRESS -------------------------------------------------- ------------------------------ ------------------------------ -------------------------------------------------- zdlras2n2.oralocal 10.160.6.3 10.160.6.4 200.254.255.24 zdlras2n1.oralocal 10.160.6.1 10.160.6.2 200.254.255.23 SQL> SQL> UPDATE rai_host SET backup_ip_address='200.254.255.23,'|| backup_ip_address WHERE node_name = 'zdlras2n1.oralocal'; 1 row updated. SQL> UPDATE rai_host SET backup_ip_address='200.254.255.24,'|| backup_ip_address WHERE node_name = 'zdlras2n2.oralocal'; 1 row updated. SQL> SELECT * FROM rai_host; NODE_NAME ADMIN_IP_ADDRESS BACKUP_IP_ADDRESS REPLICATION_IP_ADDRESS -------------------------------------------------- ------------------------------ ------------------------------ -------------------------------------------------- zdlras2n2.oralocal 10.160.6.3 200.254.255.24,10.160.6.4 200.254.255.24 zdlras2n1.oralocal 10.160.6.1 200.254.255.23,10.160.6.2 200.254.255.23 SQL> commit; Commit complete. SQL>
After that, I recommend to reboot, ate least ZDLRA database to reload the configs. Again, remember that the values here (ip, networks, interfaces, and routes) will differ for you’re your environment. Use as a guide.
Replication
The replication for ZDLRA has some steps to be done. If we do this during the deployment is easier (since is done automatically by the installer). But if we need to do after (because adding or preparing the environment) is not complicated, but we need to be aware of some details.
Be careful with a basic configuration like interface (bond or no), and ip’s. They will change in your scenario, but the most important is to check the route. The packet route needs to travel just inside the replication network because the ZDLRA configuration (at internal tables) expects/listens in just one ip.
And about ZDLRA internal change, they occur at internal tables. So, always review correctly the values to avoid errors (before the commit). After that, we can use DBMS_RA.CREATE_REPLICATION_SERVER and create the replication server config, I will show this in another post.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: ZDLRA, Creating the Replication Server | Fernando Simon