{"id":779,"date":"2020-08-03T18:16:58","date_gmt":"2020-08-03T21:16:58","guid":{"rendered":"https:\/\/www.fernandosimon.com\/blog\/?p=779"},"modified":"2020-08-03T18:16:58","modified_gmt":"2020-08-03T21:16:58","slug":"zdlra-how-to-maintain-the-native-replication","status":"publish","type":"post","link":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/","title":{"rendered":"ZDLRA, How to Maintain the Native Replication"},"content":{"rendered":"<p style=\"text-align: justify;\">The native replication for ZDLRA does not require a lot of maintenance or complicate tasks to keep it running. In my previous posts, I already wrote about <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-replication\/\" target=\"_blank\" rel=\"noopener noreferrer\">an explanation about replication<\/a>, <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-configuring-replication-network\/\" target=\"_blank\" rel=\"noopener noreferrer\">how to configure the replication network<\/a> between ZDLRA\u2019s, <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-creating-the-replication-server\/\" target=\"_blank\" rel=\"noopener noreferrer\">how to configure the replication server<\/a>, <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-creating-the-replication-config\/\" target=\"_blank\" rel=\"noopener noreferrer\">how to create the replication config<\/a> (that links everything is done before), and <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-protecting-databases-with-replication\/\" target=\"_blank\" rel=\"noopener noreferrer\">how the replication protect the database<\/a>. In this post, I will show some details that you need to monitor and to do maintain it running without errors.<\/p>\n<p style=\"text-align: justify;\"><!--more Click here to read more...--><\/p>\n<h2><a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-767 size-full\" src=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\" alt=\"\" width=\"546\" height=\"461\" srcset=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png 546w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025-300x253.png 300w\" sizes=\"auto, (max-width: 546px) 100vw, 546px\" \/><\/a><\/h2>\n<h2 style=\"text-align: justify;\">RASYS.RA_REPLICATION_SERVER<\/h2>\n<p style=\"text-align: justify;\">Table RASYS.RA_REPLICATION_SERVER contains all the information about the replication servers that were created. From there we can check which backup policies are linked with the replication and the used streams:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; col REPLICATION_SERVER_NAME format a30\r\nSQL&gt; col PROTECTION_POLICY format a20\r\nSQL&gt; col REP_SERVER_CONNECT_NAME format a20\r\nSQL&gt; set linesize 250\r\nSQL&gt; select REPLICATION_SERVER_NAME, REPLICATION_SERVER_STATE, PROTECTION_POLICY, REP_SERVER_CONNECT_NAME, MAX_STREAMS from RA_REPLICATION_SERVER;\r\n\r\nREPLICATION_SERVER_NAME        REPLICATION_SERVER_ST PROTECTION_POLICY    REP_SERVER_CONNECT_N MAX_STREAMS\r\n------------------------------ --------------------- -------------------- -------------------- -----------\r\nZDLRAS2_REP                    RUNNING               ZDLRAUP_BRONZE       RASYS                          4\r\nZDLRAS2_REP                    RUNNING               ZDLRA_BRONZE         RASYS                          4\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you can see, we have the column with useful information about the configured replication for policy. And we will have one line per replication server (that we created using the ADD_REPLICATION_SERVER as I explained here).<\/p>\n<h2 style=\"text-align: justify;\">MAX_STREAMS<\/h2>\n<p style=\"text-align: justify;\">Streams for replication servers defines the maximum of simultaneous replications tasks that the replication server can operate. This is a global definition for the entire server. So, these streams slots will be shared by all protection policies linked with the server.<\/p>\n<p style=\"text-align: justify;\">The default value is 4, but can be defined when you use the CREATE_REPLICATION_SERVER or updating later using the UPDATE_REPLICATION_SERVER:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; BEGIN\r\n  2  DBMS_RA.UPDATE_REPLICATION_SERVER (\r\n  3      replication_server_name =&gt; 'zdlras2_rep',\r\n  4      max_streams =&gt; 8);\r\n  5  END;\r\n  6  \/\r\n\r\nPL\/SQL procedure successfully completed.\r\n\r\nSQL&gt; select REPLICATION_SERVER_NAME, REPLICATION_SERVER_STATE, PROTECTION_POLICY, REP_SERVER_CONNECT_NAME, MAX_STREAMS from RA_REPLICATION_SERVER;\r\n\r\nREPLICATION_SERVER_NAME        REPLICATION_SERVER_ST PROTECTION_POLICY    REP_SERVER_CONNECT_N MAX_STREAMS\r\n------------------------------ --------------------- -------------------- -------------------- -----------\r\nZDLRAS2_REP                    RUNNING               ZDLRAUP_BRONZE       RASYS                          8\r\nZDLRAS2_REP                    RUNNING               ZDLRA_BRONZE         RASYS                          8\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you can see, changing the server affected all. The way to tune this parameter is by checking the tasks that you have in the queue. If you see that tasks are increasing in the waitlist, you can think to increase the parameter. But please remember to verify your replication network (it is not at full usage), and the size of your backupset (incremental and archivelogs). Because if they are big, increase the concurrent stream task will not help.<\/p>\n<h2 style=\"text-align: justify;\">TASKS and Library<\/h2>\n<p style=\"text-align: justify;\">Everything inside of ZDLRA is one task, <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-internals-index_backup-task-in-details\/\" target=\"_blank\" rel=\"noopener noreferrer\">INDEX_BACKUP<\/a>, <a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-osb-and-clone-to-tape\/\" target=\"_blank\" rel=\"noopener noreferrer\">tape clones<\/a>, validations, everything. And this is the same for replication. Each backupset that will be replication receives one task that is managed by upstream ZDLRA to send to downstream ZDLRA. And it is managed in the same way as a clone to tape tasks.<\/p>\n<p style=\"text-align: justify;\">The main effort is to control and checks the tasks at the upstream side. From downstream the tasks will be just index_backup type.<\/p>\n<h3 style=\"text-align: justify;\">RA_SBT_LIBRARY<\/h3>\n<p style=\"text-align: justify;\">This table store the information about the SBT library that you configured inside of ZDLRA. You will have one line per tape library and per server that you created for replication. This table is used to purely identify the LIB_KEY for your replication server (that will be used to track tasks later):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; SELECT lib_key, lib_name FROM RA_SBT_LIBRARY;\r\n\r\n   LIB_KEY LIB_NAME\r\n---------- ------------------------------\r\n     11234 OSBSBT\r\n      9044 REP$LIB_9043_ZDLRAS2_REP\r\n\r\nSQL&gt;<\/pre>\n<h3 style=\"text-align: justify;\">RA_TASK<\/h3>\n<p style=\"text-align: justify;\">With the lib_key we can search inside of RA_TASK what are the tasks related to the replication. The type will be, most of the time the tasks will be BACKUP_SBT. The query bellow report all tasks running (I took after the backup was taken and the replication is running):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task WHERE lib_key = 9044 and archived = 'N' order by 5,2,7,10,11,12,13;\r\n\r\n   TASK_ID TASK_TYPE       STATE                WAITING_ON     DB_KEY DB_UNIQUE_NAME       CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY\r\n---------- --------------- -------------------- ---------- ---------- -------------------- ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------\r\n     50581 BACKUP_SBT      COMPLETED                            25140 ORCL19               23-JUL-20 12.24.52.632234 AM +02:00           0               0                 31181\r\n     50584 BACKUP_SBT      COMPLETED                            25140 ORCL19               23-JUL-20 12.25.09.687280 AM +02:00           0               0                 31185\r\n     50586 BACKUP_SBT      COMPLETED                            25140 ORCL19               23-JUL-20 12.25.22.223941 AM +02:00           0               0                 31190\r\n     50592 BACKUP_SBT      COMPLETED                            25140 ORCL19               23-JUL-20 12.25.31.770802 AM +02:00           0               0                 31195\r\n     50595 BACKUP_SBT      EXECUTABLE                           25140 ORCL19               23-JUL-20 12.25.43.271311 AM +02:00           0               0                 31213\r\n     50596 SPAWN_SBT       EXECUTABLE                                                      23-JUL-20 12.25.43.379794 AM +02:00           0               0\r\n\r\n6 rows selected.\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you can see, you have one BACKUP_SBT per backupset. A similar query can be done using table RA_SBT_TASK.<\/p>\n<p style=\"text-align: justify;\">As usual for every ZDLRA task, the most important is always check the STATE column. If is COMPLETED, RUNNING, or EXECUTABLE is ok, but if the state is RECONCILE_WAIT this means that something is wrong with the replication (can be paused or with error) and is important to check the table ra_incident_log:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select error_text from ra_incident_log where status = 'ACTIVE' and parameter = 'RECONCILE' order by last_seen desc;\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-64771: reconcile error during DO_PENDING_REP_SETUP; replication server: ZDLRAS2_REP; database: ORCL19\r\nORA-06512: at \"SYS.DBMS_SYS_ERROR\", line 105\r\nORA-64772: downstream replication server:  is not accessible\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3468\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1809\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1763\r\nORA-28002: the password will expire within 7 days\r\nORA-06512: at \"SYS.KBRSI_ICD\", line 517\r\nORA-06512: at \"RASYS.DBMS_RAI_WALLET2URL\", line 9\r\nORA-06512: at line 1\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1723\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3364\r\n\r\n\r\nSQL&gt;<\/pre>\n<h2 style=\"text-align: justify;\">PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER<\/h2>\n<p style=\"text-align: justify;\">Sometimes is needed to pause the replication, it is a normal procedure and done using a well-documented procedure (to pause and resume the replication).<\/p>\n<p style=\"text-align: justify;\">So, to pause the replication we use DBMS_RA.PAUSE_REPLICATION_SERVER and will affect the whole server (and not just one policy):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; BEGIN\r\n  2      DBMS_RA.PAUSE_REPLICATION_SERVER (\r\n  3          replication_server_name =&gt; 'ZDLRAS2_REP'\r\n  4      );\r\n  5  END;\r\n  6  \/\r\n\r\nPL\/SQL procedure successfully completed.\r\n\r\nSQL&gt; select REPLICATION_SERVER_NAME, PROTECTION_POLICY , REPLICATION_SERVER_STATE FROM RA_REPLICATION_SERVER;\r\n\r\nREPLICATION_SERVER_N PROTECTION_POLICY    REPLICATION_SERVER_ST\r\n-------------------- -------------------- ---------------------\r\nZDLRAS2_REP          ZDLRAUP_BRONZE       PAUSED\r\nZDLRAS2_REP          ZDLRA_BRONZE         PAUSED\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you see above, the REPLICATION_SERVER_STATE column at RASYS.RA_REPLICATION_SERVER table will be as PAUSED.<\/p>\n<p style=\"text-align: justify;\">When pausing the replication, all the tasks will be put in wait and will stay in the queue until you resume the replication:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select REPLICATION_SERVER_NAME, PROTECTION_POLICY , REPLICATION_SERVER_STATE FROM RA_REPLICATION_SERVER;\r\n\r\nREPLICATION_SERVER_N PROTECTION_POLICY    REPLICATION_SERVER_ST\r\n-------------------- -------------------- ---------------------\r\nZDLRAS2_REP          ZDLRAUP_BRONZE       PAUSED\r\nZDLRAS2_REP          ZDLRA_BRONZE         PAUSED\r\n\r\nSQL&gt; select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task WHERE lib_key = 9044 and archived = 'N' order by 5,2,7,10,11,12,13;\r\n\r\n   TASK_ID TASK_TYPE       STATE           WAITING_ON     DB_KEY DB_UNIQUE_NAME       CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY\r\n---------- --------------- --------------- ---------- ---------- -------------------- ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------\r\n     50614 BACKUP_SBT      LIBRARY_WAIT                    25140 ORCL19               23-JUL-20 12.49.32.442643 AM +02:00           0               0                 31281\r\n     50617 BACKUP_SBT      LIBRARY_WAIT                    25140 ORCL19               23-JUL-20 12.49.37.325911 AM +02:00           0               0                 31285\r\n     50615 SPAWN_SBT       EXECUTABLE                                                 23-JUL-20 12.49.32.592927 AM +02:00           0               0\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">To resume the replication is easy and done with the procedure DBMS_RA.RESUME_REPLICATION_SERVER:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; BEGIN\r\n  2      DBMS_RA.RESUME_REPLICATION_SERVER (\r\n  3          replication_server_name =&gt; 'ZDLRAS2_REP'\r\n  4      );\r\n  5  END;\r\n  6  \/\r\n\r\nPL\/SQL procedure successfully completed.\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">And after that, all tasks will be in EXECUTABLE mode:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task WHERE lib_key = 9044 and archived = 'N' order by 5,2,7,10,11,12,13;\r\n\r\n   TASK_ID TASK_TYPE       STATE           WAITING_ON     DB_KEY DB_UNIQUE_NAME       CREATION_TIME                       ERROR_COUNT INTERRUPT_COUNT     BP_KEY     BS_KEY     DF_KEY     VB_KEY\r\n---------- --------------- --------------- ---------- ---------- -------------------- ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------\r\n     50614 BACKUP_SBT      RUNNING                         25140 ORCL19               23-JUL-20 12.49.32.442643 AM +02:00           0               0                 31281\r\n     50617 BACKUP_SBT      RUNNING                         25140 ORCL19               23-JUL-20 12.49.37.325911 AM +02:00           0               0                 31285\r\n     50619 BACKUP_SBT      RUNNING                         25140 ORCL19               23-JUL-20 12.49.42.598054 AM +02:00           0               0                 31289\r\n     50621 BACKUP_SBT      RUNNING                         25140 ORCL19               23-JUL-20 12.49.45.352626 AM +02:00           0               0                 31293\r\n     50623 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.49.50.271201 AM +02:00           0               0                 31313\r\n     50625 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.49.58.429705 AM +02:00           0               0                 31333\r\n     50628 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.50.00.074445 AM +02:00           0               0                 31341\r\n     50633 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.50.03.716264 AM +02:00           0               0                 31354\r\n     50635 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.50.05.911542 AM +02:00           0               0                 31358\r\n     50638 BACKUP_SBT      EXECUTABLE                      25140 ORCL19               23-JUL-20 12.50.10.922916 AM +02:00           0               0                 31382\r\n\r\n10 rows selected.\r\n\r\nSQL&gt;\r\n<\/pre>\n<h3 style=\"text-align: justify;\">RECONCILE<\/h3>\n<p style=\"text-align: justify;\">Reconcile is the process, at ZDLRA, that occurs when upstream and downstream exchange metadata and the \u201creconcile\u201d the gaps. This means that upstream will check downstream what backupssets needs to be sent. The reconcile does not occur just when you resume the replication, it is done for every ingested backup at upstream.<\/p>\n<p style=\"text-align: justify;\">If you receive errors about replication usually they will be noticed as reconcile:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select error_text from ra_incident_log where status = 'ACTIVE' and parameter = 'RECONCILE' order by last_seen desc;\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-64771: reconcile error during DO_PENDING_REP_SETUP; replication server: ZDLRAS2_REP; database: ORCL19\r\nORA-06512: at \"SYS.DBMS_SYS_ERROR\", line 105\r\nORA-64772: downstream replication server:  is not accessible\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3468\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1809\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1763\r\nORA-28002: the password will expire within 7 days\r\nORA-06512: at \"SYS.KBRSI_ICD\", line 517\r\nORA-06512: at \"RASYS.DBMS_RAI_WALLET2URL\", line 9\r\nORA-06512: at line 1\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1723\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3364\r\n\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">After you solve the problem, you can call the DBMS_RA.RESUME_REPLICATION_SERVER to do a \u201cforce\u201d restart of replication. Doing that the reconcile will be called for all replicated databases. There is a undocumented procedure (<em>dbms_ra_int.replication_reconcile(&#8216;&lt;DBNAME&gt;&#8217;,'&lt;REP_SERVER_NAME&gt;&#8217;)<\/em>) that you can call to just one database, but is not needed to use it (normal resume is enough).<\/p>\n<h3 style=\"text-align: justify;\">Replication and Patch<\/h3>\n<p style=\"text-align: justify;\">The replication for ZDLRA is basically a rman SBT tape copy (in a high, high abstraction) between two catalogs. So, when your patch is recommended that who received the replication will be at higher version.<\/p>\n<p style=\"text-align: justify;\">So, if you are using replication (whatever the mode: hub, bi-directional, etc), the basic order is:<\/p>\n<ol style=\"text-align: justify;\">\n<li>Pause replication<\/li>\n<li>Patch downstream<\/li>\n<li>Patch upstream<\/li>\n<li>Resume replication<\/li>\n<\/ol>\n<p style=\"text-align: justify;\">Of course, sometimes we can\u2019t patch both at the same time (upstream and downstream). But always pause the replication before patch downstream, and resume after patch it. This guarantee that those who received will be always in a higher version than those who send. If you need, you can resume the replication before patch the upstream to allow the reconcile and reduce the amount of data that need to be transferred (just remember to stop it before patch the upstream)<\/p>\n<p style=\"text-align: justify;\">When you operate in a bi-directional way, there is no change. You resume the replication after patch both sides (since all sides will be upstream and downstream).<\/p>\n<h2 style=\"text-align: justify;\">Know issues<\/h2>\n<p style=\"text-align: justify;\">There is a very know issue for replication and it related to the username used to replicate. This user needs to be created to be used in the wallet for DBMS_RA.CREATE_REPLICATION_SERVER (<a href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-creating-the-replication-server\/\" target=\"_blank\" rel=\"noopener noreferrer\">I explained here<\/a>). But after some time the password for this user can expire and all the replication needs to occurs any kind of warning.<\/p>\n<p style=\"text-align: justify;\">So, if you see this kind of error:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; select error_text from ra_incident_log where status = 'ACTIVE' and parameter = 'RECONCILE' order by last_seen desc;\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-64771: reconcile error during DO_PENDING_REP_SETUP; replication server: ZDLRAS2_REP; database: ORCL19\r\nORA-06512: at \"SYS.DBMS_SYS_ERROR\", line 105\r\nORA-64772: downstream replication server:  is not accessible\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3468\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1809\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1763\r\nORA-28002: the password will expire within 7 days\r\nORA-06512: at \"SYS.KBRSI_ICD\", line 517\r\nORA-06512: at \"RASYS.DBMS_RAI_WALLET2URL\", line 9\r\nORA-06512: at line 1\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 1723\r\n\r\nERROR_TEXT\r\n--------------------------------------------------------------------------------------------------------------------------\r\nORA-06512: at \"RASYS.DBMS_RCVCAT\", line 3364\r\n\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">You just need to go to downstream and reset the password for the user. You can change the profile if want too.<\/p>\n<h2 style=\"text-align: justify;\">MAINTAIN, SUSTAIN, KEEP the Replication<\/h2>\n<p style=\"text-align: justify;\">Whatever you call it, MAINTAIN, SUSTAIN, or KEEP the replication running is not difficult. The process is very similar to normal ZDLRA operation (the tasks are there and the concept is the same).<\/p>\n<p style=\"text-align: justify;\">The most important is to take care of the queue, if you see a long queue at RA_TASK table maybe you need to tune the max_streams for the server. If you already have done this, check if not reach the maximum throughput for your replication network. If you are sharing ingest and replication at the same interface you can reach a race between them and will be better to isolate them (as recommended at best practices).<\/p>\n<p><a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amogd_vm_005.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-765 size-full\" src=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amogd_vm_005.png\" alt=\"\" width=\"665\" height=\"795\" srcset=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amogd_vm_005.png 665w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amogd_vm_005-251x300.png 251w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amogd_vm_005-624x746.png 624w\" sizes=\"auto, (max-width: 665px) 100vw, 665px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p><strong>Disclaimer<\/strong>: <em>\u201cThe postings on this site are my own and don\u2019t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.<\/em> <em>Post protected by copyright.\u201d<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The native replication for ZDLRA does not require a lot of maintenance or complicate tasks to keep it running. In my previous posts, I already wrote about an explanation about replication, how to configure the replication network between ZDLRA\u2019s, how to configure the replication server, how to create the replication config (that links everything is [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[29,77,140,51],"tags":[100,65,139,74],"class_list":["post-779","post","type-post","status-publish","format-standard","hentry","category-database","category-engineeredsystems","category-replication","category-zdlra","tag-engineered-systems","tag-oracle","tag-replication","tag-zdlra"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>ZDLRA, How to Maintain the Native Replication - Fernando Simon<\/title>\n<meta name=\"description\" content=\"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ZDLRA, How to Maintain the Native Replication - Fernando Simon\" \/>\n<meta property=\"og:description\" content=\"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\" \/>\n<meta property=\"og:site_name\" content=\"Fernando Simon\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-03T21:16:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\" \/>\n<meta name=\"author\" content=\"Simon\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Simon\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\"},\"author\":{\"name\":\"Simon\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"headline\":\"ZDLRA, How to Maintain the Native Replication\",\"datePublished\":\"2020-08-03T21:16:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\"},\"wordCount\":1272,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\",\"keywords\":[\"Engineered Systems\",\"Oracle\",\"Replication\",\"ZDLRA\"],\"articleSection\":[\"Database\",\"Engineered Systems\",\"Replication\",\"ZDLRA\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\",\"name\":\"ZDLRA, How to Maintain the Native Replication - Fernando Simon\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\",\"datePublished\":\"2020-08-03T21:16:58+00:00\",\"author\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"description\":\"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\",\"contentUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fernandosimon.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ZDLRA, How to Maintain the Native Replication\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/\",\"name\":\"Fernando Simon\",\"description\":\"Have you hugged your backup today?\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\",\"name\":\"Simon\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"caption\":\"Simon\"},\"sameAs\":[\"http:\/\/www.fernandosimon.com\"],\"url\":\"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"ZDLRA, How to Maintain the Native Replication - Fernando Simon","description":"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/","og_locale":"en_US","og_type":"article","og_title":"ZDLRA, How to Maintain the Native Replication - Fernando Simon","og_description":"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.","og_url":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/","og_site_name":"Fernando Simon","article_published_time":"2020-08-03T21:16:58+00:00","og_image":[{"url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png","type":"","width":"","height":""}],"author":"Simon","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Simon","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#article","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/"},"author":{"name":"Simon","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"headline":"ZDLRA, How to Maintain the Native Replication","datePublished":"2020-08-03T21:16:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/"},"wordCount":1272,"commentCount":0,"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png","keywords":["Engineered Systems","Oracle","Replication","ZDLRA"],"articleSection":["Database","Engineered Systems","Replication","ZDLRA"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/","url":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/","name":"ZDLRA, How to Maintain the Native Replication - Fernando Simon","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage"},"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png","datePublished":"2020-08-03T21:16:58+00:00","author":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"description":"How to maintain the ZDLRA replication, monitoring tasks, max_stream, using PAUSE_REPLICATION_SERVER and RESUME_REPLICATION_SERVER.","breadcrumb":{"@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#primaryimage","url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png","contentUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2020\/07\/amagd_vm_025.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.fernandosimon.com\/blog\/zdlra-how-to-maintain-the-native-replication\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fernandosimon.com\/blog\/"},{"@type":"ListItem","position":2,"name":"ZDLRA, How to Maintain the Native Replication"}]},{"@type":"WebSite","@id":"https:\/\/www.fernandosimon.com\/blog\/#website","url":"https:\/\/www.fernandosimon.com\/blog\/","name":"Fernando Simon","description":"Have you hugged your backup today?","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9","name":"Simon","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","caption":"Simon"},"sameAs":["http:\/\/www.fernandosimon.com"],"url":"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5ofTp-cz","_links":{"self":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/779","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/comments?post=779"}],"version-history":[{"count":0,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/779\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/media?parent=779"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/categories?post=779"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/tags?post=779"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}