{"id":950,"date":"2022-08-07T18:14:30","date_gmt":"2022-08-07T21:14:30","guid":{"rendered":"https:\/\/www.fernandosimon.com\/blog\/?p=950"},"modified":"2023-12-25T19:58:55","modified_gmt":"2023-12-25T22:58:55","slug":"21c-dg-pdb","status":"publish","type":"post","link":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/","title":{"rendered":"21c, DG PDB"},"content":{"rendered":"<p style=\"text-align: justify;\">Since the 21c was public available the Data Guard per Pluggable Database &#8211; DG PDB &#8211; was intended to be there, but Oracle needed more time to make things work and some weeks ago <a href=\"https:\/\/blogs.oracle.com\/maa\/post\/dgpdb-available-today\" target=\"_blank\" rel=\"noopener\">released the feature with the 21.7 version<\/a>. Here in this post, I will show to configure it and also how to troubleshoot, and the pitfalls of using it. As usual, all the steps, logs, and outputs are covered here and I hope that it helps you understand the whole DG PDB process.<\/p>\n<h2 style=\"text-align: justify;\">My environment<\/h2>\n<p style=\"text-align: justify;\">The environment that I am using here is:<\/p>\n<ul style=\"text-align: justify;\">\n<li>Two databases running in RAC mode (two nodes in each cluster).<\/li>\n<li>ASM: same DATA and RECO diskgroups names in each cluster.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">About the databases I have:<\/p>\n<ul style=\"text-align: justify;\">\n<li>ORADBDC1, that have the pdb PDBDC1. So, they represent the DC1.<\/li>\n<li>ORADBDC2, that have the pdb PDBDC2. So, they represent the DC2.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">Each of these clusters is in a separate environment, this means that both are primary databases inside each datacenter. So, they have no DG configured between them.<\/p>\n<p style=\"text-align: justify;\">The main target for this post is to have the pdb from DC2 protected by the ORADBDC1 at the DC1. I used RAC and ASM because this is usually the normal configuration for the MAA (following the recommended architectures baseline) when using DG. This increases the protection and reduces the SPOF of your environment.<\/p>\n<h2 style=\"text-align: justify;\">DG PDB<\/h2>\n<p style=\"text-align: justify;\">The idea of DG PDB differs a little from what we see commonly for Data Guard, here each container have own life. This means that only the pdb is protected and not the entire cdb. This puts the DG PDB close to Cloud than On-Prem because it fit perfectly at the OCI structure since you can create your pdb in one region and choose another region to protect it. And even closer if you think for Autonomous Database that your ownership is pdb only. I will not say that is good or not, but is linked to how Oracle works with OCI. Personally, I prefer to have normal DG configured to protect my databases and I choose where I want to open my pdb (maybe they add this feature in the future).<\/p>\n<p style=\"text-align: justify;\">Another detail is that DG PDG (from now) works only in MaxPerformance mode, so, there is no SYNC mode for the archive destinations. There are more limitations for the DG PDB and you can check it in the topic <a href=\"https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-database\/21\/dgbkr\/managing-oracle-data-guard-broker-configurations.html#GUID-73A7F895-2DEE-41D1-9103-2EC180B68AAD\" target=\"_blank\" rel=\"noopener\">DG PDB Configuration Restrictions from official documentation<\/a> (I recommend that you read it).<\/p>\n<p><strong>Please read my new blog post about the new changes to the process. You can see how the process evolved and it is better<\/strong>. <a href=\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb-new-steps\/\" target=\"_blank\" rel=\"noopener\">Read it here.<\/a>&nbsp;<\/p>\n<p style=\"text-align: justify;\"><!--more Click here to read more...--><\/p>\n<h2 style=\"text-align: justify;\">Preparing databases<\/h2>\n<p style=\"text-align: justify;\">Starting the process, let\u2019s check what I have configured now. Basically, what is configured is the image below.<\/p>\n<p><a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-961 size-large\" src=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png\" alt=\"\" width=\"625\" height=\"293\" srcset=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png 1024w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-300x141.png 300w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-768x360.png 768w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1536x720.png 1536w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-624x293.png 624w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png 1988w\" sizes=\"auto, (max-width: 625px) 100vw, 625px\" \/><\/a>The first thing to do is to prepare both databases to meet the requirements before starting the broker configuration.<\/p>\n<h3 style=\"text-align: justify;\">Current view<\/h3>\n<p style=\"text-align: justify;\">The current view for DC1 is:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 19:50:44 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; show pdbs\r\n\r\n    CON_ID CON_NAME                       OPEN MODE  RESTRICTED\r\n---------- ------------------------------ ---------- ----------\r\n         2 PDB$SEED                       READ ONLY  NO\r\n         3 PDBDC1                         READ WRITE NO\r\nSQL&gt; show parameter db_name\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\ndb_name                              string      oradbdc1\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">And for DC2:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8sn1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 19:51:29 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; show pdbs\r\n\r\n    CON_ID CON_NAME                       OPEN MODE  RESTRICTED\r\n---------- ------------------------------ ---------- ----------\r\n         2 PDB$SEED                       READ ONLY  NO\r\n         3 PDBDC2                         READ WRITE NO\r\nSQL&gt; show parameter db_name\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\ndb_name                              string      oradbdc2\r\nSQL&gt;<\/pre>\n<h3 style=\"text-align: justify;\">TNSNAMES<\/h3>\n<p style=\"text-align: justify;\">The first is step is preparing the connections between databases. So, you need to have both entries (for each database) at your tnsnames.ora file. Bellow, you can see that both nodes (for each cluster) resolve the same entries names (this is important that each site resolve the same name):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ tnsping ORADBDC1\r\n\r\nTNS Ping Utility for Linux: Version 21.0.0.0.0 - Production on 31-JUL-2022 19:53:27\r\n\r\nCopyright (c) 1997, 2021, Oracle.  All rights reserved.\r\n\r\nUsed parameter files:\r\n\r\n\r\nUsed TNSNAMES adapter to resolve the alias\r\nAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oel8-21c-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oradbdc1)))\r\nOK (10 msec)\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ tnsping ORADBDC2\r\n\r\nTNS Ping Utility for Linux: Version 21.0.0.0.0 - Production on 31-JUL-2022 19:53:32\r\n\r\nCopyright (c) 1997, 2021, Oracle.  All rights reserved.\r\n\r\nUsed parameter files:\r\n\r\n\r\nUsed TNSNAMES adapter to resolve the alias\r\nAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oel8s-21c-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oradbdc2)))\r\nOK (0 msec)\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ scp \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/tnsnames.ora oel8n2-21c:\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/tnsnames.ora\r\ntnsnames.ora                                                                                                               100%  893   549.1KB\/s   00:00\r\n[oracle@oel8n1-21c ~]$\r\n\r\n\r\n##############################\r\n#\r\n# At DC2\r\n#\r\n##############################\r\n[oracle@oel8sn1-21c ~]$ tnsping ORADBDC1\r\n\r\nTNS Ping Utility for Linux: Version 21.0.0.0.0 - Production on 31-JUL-2022 19:53:59\r\n\r\nCopyright (c) 1997, 2021, Oracle.  All rights reserved.\r\n\r\nUsed parameter files:\r\n\r\n\r\nUsed TNSNAMES adapter to resolve the alias\r\nAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oel8-21c-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oradbdc1)))\r\nOK (10 msec)\r\n[oracle@oel8sn1-21c ~]$\r\n[oracle@oel8sn1-21c ~]$ tnsping ORADBDC2\r\n\r\nTNS Ping Utility for Linux: Version 21.0.0.0.0 - Production on 31-JUL-2022 19:54:01\r\n\r\nCopyright (c) 1997, 2021, Oracle.  All rights reserved.\r\n\r\nUsed parameter files:\r\n\r\n\r\nUsed TNSNAMES adapter to resolve the alias\r\nAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oel8s-21c-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = oradbdc2)))\r\nOK (0 msec)\r\n[oracle@oel8sn1-21c ~]$\r\n[oracle@oel8sn1-21c ~]$ scp \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/tnsnames.ora oel8sn2-21c:\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/tnsnames.ora\r\ntnsnames.ora                                                                                                               100%  893   791.1KB\/s   00:00\r\n[oracle@oel8sn1-21c ~]$<\/pre>\n<h3 style=\"text-align: justify;\">Basic database parameters and adjusts<\/h3>\n<p style=\"text-align: justify;\">To start to configure DG PDB you don\u2019t need too much, you just need to guarantee in each database the following:<\/p>\n<ul style=\"text-align: justify;\">\n<li>dg_broker_start and dg_broker_config_file* parameters configured and set.<\/li>\n<li>standby_file_management defined as AUTO.<\/li>\n<li>Standby redo log files for CDB level set.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">You can see that I made that for databases at the database at DC1:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 19:57:44 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; show parameter broker\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\nconnection_brokers                   string      ((TYPE=DEDICATED)(BROKERS=1)),\r\n                                                  ((TYPE=EMON)(BROKERS=1))\r\ndg_broker_config_file1               string      \/u01\/app\/oracle\/homes\/OraDB21H\r\n                                                 ome1\/dbs\/dr1oradbdc1.dat\r\ndg_broker_config_file2               string      \/u01\/app\/oracle\/homes\/OraDB21H\r\n                                                 ome1\/dbs\/dr2oradbdc1.dat\r\ndg_broker_start                      boolean     FALSE\r\nuse_dedicated_broker                 boolean     FALSE\r\nSQL&gt;\r\nSQL&gt;\r\nSQL&gt; alter system set dg_broker_config_file1 = '+DATA\/oradbdc1\/dr1oradbdc1.dat'  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt; alter system set dg_broker_config_file2 = '+RECO\/oradbdc1\/dr2oradbdc1.dat'  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt; alter system set dg_broker_start=TRUE  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;\r\nSQL&gt; show parameter standby_file_management\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\nstandby_file_management              string      MANUAL\r\nSQL&gt; alter system set standby_file_management = AUTO scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;\r\nSQL&gt; select group#,thread#,bytes from v$standby_log;\r\n\r\nno rows selected\r\n\r\nSQL&gt; select group#,thread#,bytes from v$log;\r\n\r\n    GROUP#    THREAD#      BYTES\r\n---------- ---------- ----------\r\n         1          1  209715200\r\n         2          1  209715200\r\n         3          2  209715200\r\n         4          2  209715200\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; select group#,thread#,bytes from v$standby_log;\r\n\r\n    GROUP#    THREAD#      BYTES\r\n---------- ---------- ----------\r\n         5          1  209715200\r\n         6          1  209715200\r\n         7          1  209715200\r\n         8          2  209715200\r\n         9          2  209715200\r\n        10          2  209715200\r\n\r\n6 rows selected.\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">And for databases at DC2 as well:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8sn1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 19:59:25 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; show parameter broker\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\nconnection_brokers                   string      ((TYPE=DEDICATED)(BROKERS=1)),\r\n                                                  ((TYPE=EMON)(BROKERS=1))\r\ndg_broker_config_file1               string      \/u01\/app\/oracle\/homes\/OraDB21H\r\n                                                 ome1\/dbs\/dr1oradbdc2.dat\r\ndg_broker_config_file2               string      \/u01\/app\/oracle\/homes\/OraDB21H\r\n                                                 ome1\/dbs\/dr2oradbdc2.dat\r\ndg_broker_start                      boolean     FALSE\r\nuse_dedicated_broker                 boolean     FALSE\r\nSQL&gt; alter system set dg_broker_config_file1 = '+DATA\/oradbdc2\/dr1oradbdc2.dat'  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;  alter system set dg_broker_config_file2 = '+RECO\/oradbdc2dr2oradbdc2.dat'  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt; alter system set dg_broker_start=TRUE  scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;\r\nSQL&gt; show parameter standby_file_management\r\n\r\nNAME                                 TYPE        VALUE\r\n------------------------------------ ----------- ------------------------------\r\nstandby_file_management              string      MANUAL\r\nSQL&gt; alter system set standby_file_management = AUTO scope = both sid = '*' ;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;\r\nSQL&gt; select group#,thread#,bytes from v$standby_log;\r\n\r\nno rows selected\r\n\r\nSQL&gt; select group#,thread#,bytes from v$log;\r\n\r\n    GROUP#    THREAD#      BYTES\r\n---------- ---------- ----------\r\n         1          1  209715200\r\n         2          1  209715200\r\n         3          2  209715200\r\n         4          2  209715200\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 ('+RECO', '+DATA') size 209715200;\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; select group#,thread#,bytes from v$standby_log;\r\n\r\n    GROUP#    THREAD#      BYTES\r\n---------- ---------- ----------\r\n         5          1  209715200\r\n         6          1  209715200\r\n         7          1  209715200\r\n         8          2  209715200\r\n         9          2  209715200\r\n        10          2  209715200\r\n\r\n6 rows selected.\r\n\r\nSQL&gt;<\/pre>\n<h3 style=\"text-align: justify;\">Remote connection and Wallet<\/h3>\n<p style=\"text-align: justify;\">To allow one database to register another for DG, you need to allow remote connection (user with SYSDBA\/SYSDG permissions). And here we have the first pitfall.<\/p>\n<p style=\"text-align: justify;\">The only way that I made it work was using the wallet and this is due to the way that the configuration is made inside the broker. In the end, I have the <strong>ISSUE #1<\/strong> topic that describes with more information.<\/p>\n<p style=\"text-align: justify;\">What you need to know and understand is that the databases will connect using a defined \u201c<em>connect identifier<\/em>\u201d and this needs to be one tns entry or ezconnect. A<strong>nd when using a wallet, the credential name inside of the wallet needs to have the same name as your tns entry for each database<\/strong>. So, the way to prepare this is exactly like this:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ mkdir -p \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ mkstore -wrl \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc -createALO\r\nOracle Secret Store Tool Release 21.0.0.0.0 - Production\r\nVersion 21.0.0.0.0\r\nCopyright (c) 2004, 2022, Oracle and\/or its affiliates. All rights reserved.\r\n\r\n[oracle@oel8n1-21c ~]$ mkstore -wrl \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc -createCredential ORADBDC1 sys oracle\r\nOracle Secret Store Tool Release 21.0.0.0.0 - Production\r\nVersion 21.0.0.0.0\r\nCopyright (c) 2004, 2022, Oracle and\/or its affiliates. All rights reserved.\r\n\r\n[oracle@oel8n1-21c ~]$ mkstore -wrl \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc -createCredential ORADBDC2 sys oracle\r\nOracle Secret Store Tool Release 21.0.0.0.0 - Production\r\nVersion 21.0.0.0.0\r\nCopyright (c) 2004, 2022, Oracle and\/or its affiliates. All rights reserved.\r\n\r\n[oracle@oel8n1-21c ~]$ mkstore -wrl \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc -listCredential\r\nOracle Secret Store Tool Release 21.0.0.0.0 - Production\r\nVersion 21.0.0.0.0\r\nCopyright (c) 2004, 2022, Oracle and\/or its affiliates. All rights reserved.\r\n\r\nList credential (index: connect_string username)\r\n2: ORADBDC2 sys\r\n1: ORADBDC1 sys\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ ls -l \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\ntotal 4\r\n-rw------- 1 oracle oinstall 941 Aug  2 21:33 cwallet.sso\r\n-rw------- 1 oracle oinstall   0 Aug  2 21:32 cwallet.sso.lck\r\n[oracle@oel8n1-21c ~]$<\/pre>\n<p style=\"text-align: justify;\">Above you can see that I made the following (in order):<\/p>\n<ol style=\"text-align: justify;\">\n<li>Created the one autologin (ALO) wallet.<\/li>\n<li>Added the credentials (for the sys users).<\/li>\n<\/ol>\n<p style=\"text-align: justify;\">After I need to configure the sqlnet.ora file (because the RFS processes &#8211; and others) will use this wallet to send the redo:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ ls -l \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\nls: cannot access '\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora': No such file or directory\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ vi \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ cat \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\nNAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)\r\n\r\nSQLNET.WALLET_OVERRIDE = true\r\n\r\nWALLET_LOCATION =\r\n(\r\n   SOURCE =\r\n      (METHOD = FILE)\r\n      (METHOD_DATA =\r\n         (DIRECTORY = \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc)\r\n      )\r\n)\r\n[oracle@oel8n1-21c ~]$<\/pre>\n<p style=\"text-align: justify;\">I recommend you to test the wallet usage with sqlplus connections before you continue. Below are some examples of how to do that, you can see that I do not specify username and password, just the tns entry (and this entry is the credential name inside the wallet).<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus \/@oradbdc2 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Tue Aug 2 21:37:33 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8n1-21c ~]$ sqlplus \/@oradbdc1 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Tue Aug 2 21:37:39 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8n1-21c ~]$<\/pre>\n<p style=\"text-align: justify;\">After that I copy the wallet and sqlnet to the other node of DC1 and to all nodes for the DC2 database:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n2-21c ~]$ mkdir \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\n[oracle@oel8n2-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\/* \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\ncwallet.sso                                                                                                            100%  941    58.2KB\/s   00:00\r\ncwallet.sso.lck                                                                                                        100%    0     0.0KB\/s   00:00\r\n[oracle@oel8n2-21c ~]$\r\n[oracle@oel8n2-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\nsqlnet.ora                                                                                                             100%  221    42.9KB\/s   00:00\r\n[oracle@oel8n2-21c ~]$\r\n\r\n##############################\r\n#\r\n# At DC2\r\n#\r\n##############################\r\n[oracle@oel8sn1-21c ~]$ mkdir -p \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\n[oracle@oel8sn1-21c ~]$\r\n[oracle@oel8sn1-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\/* \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\noracle@oel8n1-21c's password:\r\ncwallet.sso                                                                                                                100%  941   582.1KB\/s   00:00\r\ncwallet.sso.lck                                                                                                            100%    0     0.0KB\/s   00:00\r\n[oracle@oel8sn1-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\noracle@oel8n1-21c's password:\r\nsqlnet.ora                                                                                                                 100%  221    63.3KB\/s   00:00\r\n[oracle@oel8sn1-21c ~]$\r\n[oracle@oel8sn1-21c ~]$ ssh oel8sn2-21c.oralocal\r\nActivate the web console with: systemctl enable --now cockpit.socket\r\n\r\nLast login: Wed Aug  3 10:02:21 2022\r\n[oracle@oel8sn2-21c ~]$ mkdir -p \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\n[oracle@oel8sn2-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\/* \/u01\/app\/oracle\/dbs\/dgwallet_oradbdc\r\noracle@oel8n1-21c's password:\r\ncwallet.sso                                                                                                                100%  941   253.4KB\/s   00:00\r\ncwallet.sso.lck                                                                                                            100%    0     0.0KB\/s   00:00\r\n[oracle@oel8sn2-21c ~]$ scp oel8n1-21c:\/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora \/u01\/app\/oracle\/homes\/OraDB21Home1\/network\/admin\/sqlnet.ora\r\noracle@oel8n1-21c's password:\r\nsqlnet.ora                                                                                                                 100%  221   173.8KB\/s   00:00\r\n[oracle@oel8sn2-21c ~]$<\/pre>\n<p style=\"text-align: justify;\">As you can see above, I set tnsnames.ora, sqlnet.ora, and wallet outside the Oracle Home. This is because of the ROOH (Read Only Oracle Home) for 21c (as I explained in the <a href=\"https:\/\/www.fernandosimon.com\/blog\/21c-rooh\/\" target=\"_blank\" rel=\"noopener\">previous post<\/a>).<\/p>\n<p style=\"text-align: justify;\">Unfortunately, after you configure the wallet and sqlnet entry you need to restart the databases (both instances). This is needed because the current database processes started before you configure and they don\u2019t know the entries that you defined.<\/p>\n<h2 style=\"text-align: justify;\">Broker and DGMGRL<\/h2>\n<h3 style=\"text-align: justify;\">Creating configuration<\/h3>\n<p style=\"text-align: justify;\">After adjusting the basic requirements we can start to configure the Broker. So, connecting at the first node of my cluster database ORADBDC1 at DC1 I created the configuration for the local database:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ dgmgrl sys\/oracle\r\nDGMGRL for Linux: Release 21.0.0.0.0 - Production on Wed Aug 3 23:10:15 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2021, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nWelcome to DGMGRL, type \"help\" for information.\r\nConnected to \"oradbdc1\"\r\nConnected as SYSDBA.\r\nDGMGRL&gt;\r\nDGMGRL&gt; show configuration;\r\nORA-16596: member not part of the Oracle Data Guard broker configuration\r\n\r\nConfiguration details cannot be determined by DGMGRL\r\nDGMGRL&gt;\r\nDGMGRL&gt;\r\nDGMGRL&gt; create configuration dc1 primary database is oradbdc1 connect identifier is oradbdc1;\r\nConnected to \"oradbdc1\"\r\nConfiguration \"dc1\" created with primary database \"oradbdc1\"\r\nDGMGRL&gt;\r\nDGMGRL&gt;\r\nDGMGRL&gt; show configuration verbose;\r\n\r\nConfiguration - dc1\r\n\r\n  Protection Mode: MaxPerformance\r\n  Members:\r\n  oradbdc1 - Primary database\r\n\r\n  Properties:\r\n    FastStartFailoverThreshold      = '30'\r\n    OperationTimeout                = '30'\r\n    TraceLevel                      = 'USER'\r\n    FastStartFailoverLagLimit       = '30'\r\n    CommunicationTimeout            = '180'\r\n    ObserverReconnect               = '0'\r\n    ObserverPingInterval            = '0'\r\n    ObserverPingRetry               = '0'\r\n    FastStartFailoverAutoReinstate  = 'TRUE'\r\n    FastStartFailoverPmyShutdown    = 'TRUE'\r\n    BystandersFollowRoleChange      = 'ALL'\r\n    ObserverOverride                = 'FALSE'\r\n    ExternalDestination1            = ''\r\n    ExternalDestination2            = ''\r\n    PrimaryLostWriteAction          = 'CONTINUE'\r\n    ConfigurationWideServiceName    = 'oradbdc1_CFG'\r\n    ConfigurationSimpleName         = 'dc1'\r\n    DrainTimeout                    = '0'\r\n\r\nFast-Start Failover:  Disabled\r\n\r\nConfiguration Status:\r\nDISABLED\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you saw, the basic configuration informs that this database is a primary database and the current connect identifier. Is the same that you do for a normal DG.<\/p>\n<p style=\"text-align: justify;\"><strong>The same needs to be done for the other site<\/strong>. So, connecting at the first node of my cluster database ORADBDC2 at DC2 I created the configuration for the local database as well:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8sn1-21c ~]$ dgmgrl sys\/oracle\r\nDGMGRL for Linux: Release 21.0.0.0.0 - Production on Wed Aug 3 23:11:17 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2021, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nWelcome to DGMGRL, type \"help\" for information.\r\nConnected to \"oradbdc2\"\r\nConnected as SYSDBA.\r\nDGMGRL&gt;\r\nDGMGRL&gt; create configuration dc2 primary database is oradbdc2 connect identifier is oradbdc2;\r\nConnected to \"oradbdc2\"\r\nConfiguration \"dc2\" created with primary database \"oradbdc2\"\r\nDGMGRL&gt;\r\nDGMGRL&gt;\r\nDGMGRL&gt; show configuration verbose;\r\n\r\nConfiguration - dc2\r\n\r\n  Protection Mode: MaxPerformance\r\n  Members:\r\n  oradbdc2 - Primary database\r\n\r\n  Properties:\r\n    FastStartFailoverThreshold      = '30'\r\n    OperationTimeout                = '30'\r\n    TraceLevel                      = 'USER'\r\n    FastStartFailoverLagLimit       = '30'\r\n    CommunicationTimeout            = '180'\r\n    ObserverReconnect               = '0'\r\n    ObserverPingInterval            = '0'\r\n    ObserverPingRetry               = '0'\r\n    FastStartFailoverAutoReinstate  = 'TRUE'\r\n    FastStartFailoverPmyShutdown    = 'TRUE'\r\n    BystandersFollowRoleChange      = 'ALL'\r\n    ObserverOverride                = 'FALSE'\r\n    ExternalDestination1            = ''\r\n    ExternalDestination2            = ''\r\n    PrimaryLostWriteAction          = 'CONTINUE'\r\n    ConfigurationWideServiceName    = 'oradbdc2_CFG'\r\n    ConfigurationSimpleName         = 'dc2'\r\n    DrainTimeout                    = '0'\r\n\r\nFast-Start Failover:  Disabled\r\n\r\nConfiguration Status:\r\nDISABLED\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\"><strong>So, now we have both sites with their own configuration. As you can see above, both configurations DC1 and DC2 appear as primary.<\/strong><\/p>\n<h3 style=\"text-align: justify;\">Add configuration<\/h3>\n<p style=\"text-align: justify;\">After that, we can use the new command for 21.7 and add one site to the other configuration. <strong>Here I am at DC1 and would add the configuration defined at DC2<\/strong>:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; add configuration dc2 connect identifier is oradbdc2;\r\nConfiguration dc2 added.\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">This is the new command for DG PDB. It allows you to add the configuration (from a remote site) to the current broker config. And here is the reason that you need to have a wallet (or somehow remote connection without requesting a password). At the moment the password is not possible to be specified (but supposed to connect using the current used to connect at dgmgrl).&nbsp; You can see at <strong>ISSUE #1<\/strong> at the end of the page that I tried several ways and nothing worked.<\/p>\n<p style=\"text-align: justify;\">As explained before, the \u201c<em>connect identifier<\/em>\u201d refers to tns entry and needs to be the same that exists in your wallet. With that, the connection can be done and the configuration can be added. After that you can see that I have both configurations identified as Primary Databases:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; show configuration verbose;\r\n\r\nConfiguration - dc1\r\n\r\n  Protection Mode: MaxPerformance\r\n  Members:\r\n  oradbdc1 - Primary database\r\n  oradbdc2 - Primary database in dc2 configuration\r\n\r\n  Properties:\r\n    FastStartFailoverThreshold      = '30'\r\n    OperationTimeout                = '30'\r\n    TraceLevel                      = 'USER'\r\n    FastStartFailoverLagLimit       = '30'\r\n    CommunicationTimeout            = '180'\r\n    ObserverReconnect               = '0'\r\n    ObserverPingInterval            = '0'\r\n    ObserverPingRetry               = '0'\r\n    FastStartFailoverAutoReinstate  = 'TRUE'\r\n    FastStartFailoverPmyShutdown    = 'TRUE'\r\n    BystandersFollowRoleChange      = 'ALL'\r\n    ObserverOverride                = 'FALSE'\r\n    ExternalDestination1            = ''\r\n    ExternalDestination2            = ''\r\n    PrimaryLostWriteAction          = 'CONTINUE'\r\n    ConfigurationWideServiceName    = 'oradbdc1_CFG'\r\n    ConfigurationSimpleName         = 'dc1'\r\n    DrainTimeout                    = '0'\r\n\r\nFast-Start Failover:  Disabled\r\n\r\nConfiguration Status:\r\nDISABLED\r\n\r\nDGMGRL&gt;\r\nDGMGRL&gt; show configuration verbose dc2\r\n\r\nConfiguration - dc2\r\n\r\nPrimary Database: oradbdc2\r\n\r\n  Properties:\r\n    DGConnectIdentifier             = ''\r\n    LogShipping                     = 'ON'\r\n    LogXptMode                      = 'ASYNC'\r\n    DelayMins                       = '0'\r\n    Binding                         = 'optional'\r\n    MaxFailure                      = '0'\r\n    ReopenSecs                      = '300'\r\n    NetTimeout                      = '30'\r\n    RedoCompression                 = 'DISABLE'\r\n\r\nConfiguration dc2 Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">After that we can enable the configuration (if you do not do that you can\u2019t add the pdb and will receive the error <em>DGM-17381: Configuration XXXX is disabled<\/em>):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; enable configuration all;\r\nSucceeded.\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">Using the ALL, the configuration is enabled on both sites.<\/p>\n<h3 style=\"text-align: justify;\">Add Pluggable Database<\/h3>\n<p style=\"text-align: justify;\">After enabling we can use the next new command for 21.7. It is the ADD PLUGGABLE DATABASE, and (as the name suggests) it adds the pdb to your broker configuration:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; ADD PLUGGABLE DATABASE pdbdc2 AT oradbdc1 SOURCE IS pdbdc2 AT oradbdc2 PDBFILENAMECONVERT IS \"'oradbdc2','oradbdc1'\";\r\nConnected to \"oradbdc2\"\r\nEnter password for DGPDB@oradbdc2:\r\nConnected to \"oradbdc1\"\r\nEnter password for DGPDB@oradbdc1:\r\nORA-17628: Oracle error 11402 returned by remote Oracle server\r\nORA-11402: Oracle Data Guard site entry not found\r\n\r\n\r\nFailed.\r\nDGMGRL&gt; ADD PLUGGABLE DATABASE pdbdc2 AT oradbdc1 SOURCE IS pdbdc2 AT oradbdc2 PDBFILENAMECONVERT IS \"'oradbdc2','oradbdc1'\";\r\nConnected to \"oradbdc2\"\r\nConnected to \"oradbdc1\"\r\n\r\nPluggable Database \"PDBDC2\" added\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">First, for the \u201cfailed\u201d, I explain <strong>ISSUE #2<\/strong> at the end of the post. But I just run the same command and the pdb is added correctly. Check that the password is not asked because the first execution already configured everything at pdb level (DGPDB_INT user).<\/p>\n<p style=\"text-align: justify;\">So, the command has some details:<\/p>\n<ul style=\"text-align: justify;\">\n<li>The pdb name: As you saw above, I added the PDBDC2 with the same name, but you can specify any name that you want. One detail is that if you change the name of pdb and do not use OFA (like using at filesystem), remember to add this name change at the pdbfilenameconvert too.<\/li>\n<li>AT: The first \u201cAT\u201d indicates the target (where) the pdb will be added. In my scenario means that I am adding it to oradbdc1 database.<\/li>\n<li>SOURCE IS: This means the source of the PDB, here it is the PDBDC2 from the oradbdc2 database.<\/li>\n<li>PDBFILENAMECONVER: It defines the transcription for datafile conversion. It works the same way as <em>db_file_name_convert<\/em>. Here I just converted the names of the databases since both sites use ASM with the same diskgroup name. But if needed you can add the \u201ccouples\u201d that you need.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">When you execute the command it will request you to define the password for the DGPDB user. Please set the same password for both. This is a new used to sync and made the connections between sites. When you add the broker check if the user exists and it is unlocked. If not, it creates or unlocks it if needed. More info at the <a href=\"https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-database\/21\/dgbkr\/managing-oracle-data-guard-broker-configurations.html#GUID-F484095B-7DBA-46AB-8867-EF8A960EB3DF\" target=\"_blank\" rel=\"noopener\">official documentation<\/a>.<\/p>\n<p style=\"text-align: justify;\">While after adding the pdb you can use the command to show the details of pdb inside of broker:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; show pluggable database pdbdc2 at oradbdc1;\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc1'\r\n\r\n  Data Guard Role:     Physical Standby\r\n  Con_ID:              5\r\n  Source:              con_id 3 at oradbdc2\r\n  Transport Lag:       0 seconds (computed 43 seconds ago)\r\n  Intended State:      APPLY-ON\r\n  Apply State:         Not Running\r\n\r\nPluggable Database Status:\r\nDGM-5103: one or more data files were not found\r\nORA-16766: Redo Apply is stopped\r\n\r\nDGMGRL&gt; show pluggable database pdbdc2 at oradbdc2;\r\nConnected to \"oradbdc2\"\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc2'\r\n\r\n  Data Guard Role:     Primary\r\n  Con_ID:              4\r\n  Active Target:       con_id 5 at oradbdc1\r\n\r\nPluggable Database Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">So, above you can see two commands, the output is different between them because at DC2 it is the primary site for the pdb, and while for DC1 it is the standby site for the pdb. Important to notice that it shows you that the CON_ID from each site is different. In my case at DC2, it is 4, and at DC1 it is 5.<\/p>\n<p style=\"text-align: justify;\">But as you saw above the command reports <em>Warning: ORA-16910: Inconsistency detected for one or more pluggable databases<\/em>. And this is normal because (apparently, and until now) the \u201cadd pluggable database\u201d does not copy the data from one site to another (<a href=\"https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-database\/21\/dgbkr\/examples-using-data-guard-broker-dgpdb-DGMGRL-utility.html#GUID-B8C509B1-C56C-4623-BF47-E2DDCEBD5BAB\" target=\"_blank\" rel=\"noopener\">directly from the documentation<\/a>: \u201c<em>After you add the source PDB, and before you start recovery on the target PDB, ensure that the database files that correspond to the source PDB are copied to the target database. Use RMAN or operating system copy commands to instantiate source PDB files.<\/em>\u201d). Adding more this is the output from my alertlog (at the target DC1) shows that files were not copied:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">2022-08-03T23:13:02.797847+02:00\r\ncreate pluggable database PDBDC2 as standby from PDBDC2 at oradbdc2 file_name_convert=('oradbdc2','oradbdc1')\r\n2022-08-03T23:13:10.850260+02:00\r\n****************************************************************\r\nPluggable Database PDBDC2 with pdb id - 5 is created as UNUSABLE.\r\nIf any errors are encountered before the pdb is marked as NEW,\r\nthen the pdb must be dropped\r\nlocal undo-1, localundoscn-0x000000000000011d\r\n****************************************************************\r\n2022-08-03T23:13:10.926824+02:00\r\nPDBDC2(5):ALTER SYSTEM SET _dgpdb_file_name_convert='oradbdc2','oradbdc1' SCOPE=SPFILE PDB='PDBDC2';\r\n*****************************************\r\nWARNING: The converted filename '+DATA\/oradbdc1\/e55cef3f4eca51d8e0535a0aa00aa405\/datafile\/system.302.1111789169'\r\n         is an ASM fully qualified filename.\r\n         Changing the filename to '+DATA\/MUST_RENAME_THIS_DATAFILE_15.4294967295.4294967295'.\r\n         Please rename it accordingly.\r\n*****************************************\r\n*****************************************\r\nWARNING: The converted filename '+DATA\/oradbdc1\/e55cef3f4eca51d8e0535a0aa00aa405\/datafile\/sysaux.300.1111789169'\r\n         is an ASM fully qualified filename.\r\n         Changing the filename to '+DATA\/MUST_RENAME_THIS_DATAFILE_16.4294967295.4294967295'.\r\n         Please rename it accordingly.\r\n*****************************************\r\n*****************************************\r\nWARNING: The converted filename '+DATA\/oradbdc1\/e55cef3f4eca51d8e0535a0aa00aa405\/datafile\/undotbs1.301.1111789169'\r\n         is an ASM fully qualified filename.\r\n         Changing the filename to '+DATA\/MUST_RENAME_THIS_DATAFILE_17.4294967295.4294967295'.\r\n         Please rename it accordingly.\r\n*****************************************\r\n*****************************************\r\nWARNING: The converted filename '+DATA\/oradbdc1\/e55cef3f4eca51d8e0535a0aa00aa405\/datafile\/undo_2.305.1111789227'\r\n         is an ASM fully qualified filename.\r\n         Changing the filename to '+DATA\/MUST_RENAME_THIS_DATAFILE_18.4294967295.4294967295'.\r\n         Please rename it accordingly.\r\n*****************************************\r\nCompleted: create pluggable database PDBDC2 as standby from PDBDC2 at oradbdc2 file_name_convert=('oradbdc2','oradbdc1')\r\n2022-08-03T23:13:13.059159+02:00<\/pre>\n<p style=\"text-align: justify;\">So, since I am using ASM as a storage system the datafiles can\u2019t even be copied (directly) because (by definition) is a fully qualified name and the direct name is not allowed.<\/p>\n<h3 style=\"text-align: justify;\">Copying Datafiles<\/h3>\n<p style=\"text-align: justify;\">So, there are several options to copy datafiles between sites. Online, and offline modes too. I prefer the online mode using rman because the interference is less over the system (and is easier to parallel things).<\/p>\n<p style=\"text-align: justify;\"><strong>I recommend to you do add ISSUE #3 at the end of the post to show some details on why I used the auxiliary at rman.<\/strong><\/p>\n<p style=\"text-align: justify;\">The first thing that you need to do is disable the configuration to avoid any type\/kind of lock over the redos\/files\/controlfile (is not clear in the documentation if this is needed):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">DGMGRL&gt; disable configuration all;\r\nSucceeded.\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">Before copying the first thing that you need to do is connect to the primary site for this pdb and discover which file numbers it has:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8sn1-21c ~]$ sqlplus sys\/oracle@oradbdc2 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Wed Aug 3 19:46:28 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; show pdbs\r\n\r\n    CON_ID CON_NAME                       OPEN MODE  RESTRICTED\r\n---------- ------------------------------ ---------- ----------\r\n         2 PDB$SEED                       READ ONLY  NO\r\n         4 PDBDC2                         READ WRITE NO\r\nSQL&gt;\r\nSQL&gt; set linesize 255\r\nSQL&gt; col file_name format a100\r\nSQL&gt; select file_id, file_name from cdb_data_files where con_id = 4;\r\n\r\n   FILE_ID FILE_NAME\r\n---------- ----------------------------------------------------------------------------------------------------\r\n        15 +DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/system.302.1111789169\r\n        16 +DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/sysaux.300.1111789169\r\n        17 +DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undotbs1.301.1111789169\r\n        18 +DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undo_2.305.1111789227\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">So, now I know that I need to copy datafiles 15 to 18 to DC1. To do that I used the command below while connected at the first node of DC1:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ rman target sys\/oracle@oradbdc2 auxiliary sys\/oracle@oradbdc1\r\n\r\nRecovery Manager: Release 21.0.0.0.0 - Production on Wed Aug 3 23:20:27 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2021, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nconnected to target database: ORADBDC2 (DBID=3458508890)\r\nconnected to auxiliary database: ORADBDC1 (DBID=2977563181)\r\n\r\nRMAN&gt;\r\n\r\nRMAN&gt; run {\r\n2&gt; allocate channel d1 type disk ;\r\n3&gt; backup as copy reuse  datafile 15,16,17,18 auxiliary format '+DATA';\r\n4&gt; }\r\n\r\nusing target database control file instead of recovery catalog\r\nallocated channel: d1\r\nchannel d1: SID=69 instance=oradbdc22 device type=DISK\r\n\r\nStarting backup at 03-AUG-22\r\nchannel d1: starting datafile copy\r\ninput datafile file number=00016 name=+DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/sysaux.300.1111789169\r\noutput file name=+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/sysaux.316.1111792861 tag=TAG20220803T232100\r\nchannel d1: datafile copy complete, elapsed time: 00:00:25\r\nchannel d1: starting datafile copy\r\ninput datafile file number=00015 name=+DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/system.302.1111789169\r\noutput file name=+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/system.317.1111792887 tag=TAG20220803T232100\r\nchannel d1: datafile copy complete, elapsed time: 00:00:15\r\nchannel d1: starting datafile copy\r\ninput datafile file number=00017 name=+DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undotbs1.301.1111789169\r\noutput file name=+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undotbs1.318.1111792901 tag=TAG20220803T232100\r\nchannel d1: datafile copy complete, elapsed time: 00:00:03\r\nchannel d1: starting datafile copy\r\ninput datafile file number=00018 name=+DATA\/ORADBDC2\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undo_2.305.1111789227\r\noutput file name=+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undo_2.319.1111792905 tag=TAG20220803T232100\r\nchannel d1: datafile copy complete, elapsed time: 00:00:03\r\nFinished backup at 03-AUG-22\r\nreleased channel: d1\r\n\r\nRMAN&gt;<\/pre>\n<p style=\"text-align: justify;\">Above you can see that i:<\/p>\n<ul style=\"text-align: justify;\">\n<li>My target is the oradbdc2 at DC2.<\/li>\n<li>Auxiliary is the oradbdc1 at DC1.<\/li>\n<li>BACKUP AS COPY, so, the datafiles specified will be copied AS IS.<\/li>\n<li>AUXILIARY FORMAT defined the destination (so I don\u2019t need to specify newname since is done automatically).<\/li>\n<li>From the output, you can see the source and target datafilenames (and the conversion as well).<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">So, at ASM in DC1 now I have this:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">ASMCMD&gt; ls -l E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\r\nType      Redund  Striped  Time             Sys  Name\r\nDATAFILE  UNPROT  COARSE   AUG 03 23:00:00  Y    SYSAUX.316.1111792861\r\nDATAFILE  UNPROT  COARSE   AUG 03 23:00:00  Y    SYSTEM.317.1111792887\r\nDATAFILE  UNPROT  COARSE   AUG 03 23:00:00  Y    UNDOTBS1.318.1111792901\r\nDATAFILE  UNPROT  COARSE   AUG 03 23:00:00  Y    UNDO_2.319.1111792905\r\nASMCMD&gt;<\/pre>\n<p style=\"text-align: justify;\">As you saw previously the output from my alertlog the datafiles names was registered with \u201cMUST_RENAME\u201d at the controlfile. This can be checked directly at database oradbdc1 at DC1:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Wed Aug 3 23:22:23 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; \r\nSQL&gt; col NAME format a50\r\nSQL&gt; col VALUE$ format a50\r\nSQL&gt; select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ ;\r\n\r\nDB_UNIQ_NAME                   NAME                                               VALUE$                                                PDB_UID\r\n------------------------------ -------------------------------------------------- -------------------------------------------------- ----------\r\n*                              _dgpdb_file_name_convert                           'oradbdc2','oradbdc1'                              565232866\r\n\r\nSQL&gt;\r\nSQL&gt; show pdbs\r\n\r\n    CON_ID CON_NAME                       OPEN MODE  RESTRICTED\r\n---------- ------------------------------ ---------- ----------\r\n         2 PDB$SEED                       READ ONLY  NO\r\n         3 PDBDC1                         READ WRITE NO\r\n         5 PDBDC2                         MOUNTED\r\nSQL&gt; alter session set container = PDBDC2;\r\n\r\nSession altered.\r\n\r\nSQL&gt; set linesize 255\r\nSQL&gt; col name format a70\r\nSQL&gt; select file#, name from v$datafile;\r\n\r\n     FILE# NAME\r\n---------- ----------------------------------------------------------------------\r\n        85 +DATA\/MUST_RENAME_THIS_DATAFILE_15.4294967295.4294967295\r\n        86 +DATA\/MUST_RENAME_THIS_DATAFILE_16.4294967295.4294967295\r\n        87 +DATA\/MUST_RENAME_THIS_DATAFILE_17.4294967295.4294967295\r\n        88 +DATA\/MUST_RENAME_THIS_DATAFILE_18.4294967295.4294967295\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">&nbsp;<\/p>\n<p style=\"text-align: justify;\">So, to fix that we need to rename the datafiles. This is done using the rename command and specifying the source and target files. <strong>Please take attention to rename correctly the datafile names to the correspondent ones<\/strong>:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">SQL&gt; alter database rename file '+DATA\/MUST_RENAME_THIS_DATAFILE_15.4294967295.4294967295' to '+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/system.317.1111792887';\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; alter database rename file '+DATA\/MUST_RENAME_THIS_DATAFILE_16.4294967295.4294967295' to '+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/sysaux.316.1111792861';\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; alter database rename file '+DATA\/MUST_RENAME_THIS_DATAFILE_17.4294967295.4294967295' to '+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undotbs1.318.1111792901';\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; alter database rename file '+DATA\/MUST_RENAME_THIS_DATAFILE_18.4294967295.4294967295' to '+DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undo_2.319.1111792905';\r\n\r\nDatabase altered.\r\n\r\nSQL&gt; col  name format a100\r\nSQL&gt; select file#, name from v$datafile;\r\n\r\n     FILE# NAME\r\n---------- ----------------------------------------------------------------------------------------------------\r\n        85 +DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/system.317.1111792887\r\n        86 +DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/sysaux.316.1111792861\r\n        87 +DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undotbs1.318.1111792901\r\n        88 +DATA\/ORADBDC1\/E55CEF3F4ECA51D8E0535A0AA00AA405\/DATAFILE\/undo_2.319.1111792905\r\n\r\nSQL&gt;<\/pre>\n<h3 style=\"text-align: justify;\">Enable Configuration<\/h3>\n<p style=\"text-align: justify;\">After copying the datafiles and renaming them, we can come back to dgmgrl and enable the configuration again. Doing that the RFS starts again the redo start to apply correctly:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; enable configuration all;\r\nSucceeded.\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">And if you look at alertlog from oradbdc1 (that is the standby for the pdb) you can see that the redo is in transit and the \u201c<em>alter pluggable database pdbdc2 recover managed standby database<\/em>\u201d is in place:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">2022-08-03T23:28:01.283876+02:00\r\nALTER SYSTEM SET fal_server='oradbdc2' SCOPE=BOTH;\r\nalter pluggable database PDBDC2 recover managed standby database disconnect\r\nalter pluggable database \"PDBDC2\" close instances=all\r\nCompleted: alter pluggable database \"PDBDC2\" close instances=all\r\n2022-08-03T23:28:01.333398+02:00\r\nRSM0 (PID:406236): Requesting Managed Recovery process for PDBID:5 [krsm.c:1352]\r\n2022-08-03T23:28:01.347156+02:00\r\nTT04 (PID:489392): Background Managed Recovery process started [krsm.c:1697]\r\n2022-08-03T23:28:01.355674+02:00\r\nPDBDC2(5):Serial Media Recovery started\r\n2022-08-03T23:28:01.376770+02:00\r\n.... (PID:405802): PDBID:5 Managed Recovery starting Real Time Apply [krsm.c:15931]\r\n2022-08-03T23:28:01.428582+02:00\r\nPDBDC2(5):max_pdb is 5\r\n2022-08-03T23:28:01.568393+02:00\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Log +RECO\/ORADBDC1\/ARCHIVELOG\/2022_08_03\/thread_1_seq_47.489.1111792393 [krd.c:9452]\r\n2022-08-03T23:28:01.874258+02:00\r\nALTER SYSTEM SET remote_listener=' oel8-21c-scan:1521' SCOPE=MEMORY SID='oradbdc11';\r\n2022-08-03T23:28:01.879113+02:00\r\nALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='oradbdc11';\r\n2022-08-03T23:28:02.335897+02:00\r\nCompleted: alter pluggable database PDBDC2 recover managed standby database disconnect\r\n2022-08-03T23:28:03.060207+02:00\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Log +RECO\/ORADBDC1\/ARCHIVELOG\/2022_08_03\/thread_2_seq_39.488.1111792393 [krd.c:9452]\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Waiting for T-1.S-48 (in transit) [krsm.c:6205]<\/pre>\n<p style=\"text-align: justify;\">And when the archivelolg is generated at the primary site of the pdb, it is applied correctly at the standby:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">##############################\r\n#\r\n# At DC2\r\n#\r\n##############################\r\n\r\nSQL&gt; alter system archive log current;\r\n\r\nSystem altered.\r\n\r\nSQL&gt;\r\n\r\n##############################\r\n#\r\n# Alertlog at DC2\r\n#\r\n##############################\r\n2022-08-03T23:28:37.724487+02:00\r\nALTER SYSTEM ARCHIVE LOG\r\n2022-08-03T23:28:38.460517+02:00\r\nNET  (PID:512199): Archived Log entry 130 added for B-1111517533.T-2.S-40 ID 0x7f48ce247857 LAD:1 [krse.c:4934]\r\n2022-08-03T23:28:38.628458+02:00\r\nThread 1 advanced to log sequence 49 (LGWR switch),  current SCN: 11795760\r\n  Current log# 1 seq# 49 mem# 0: +DATA\/ORADBDC2\/ONLINELOG\/group_1.289.1111517533\r\n  Current log# 1 seq# 49 mem# 1: +RECO\/ORADBDC2\/ONLINELOG\/group_1.308.1111517543\r\n2022-08-03T23:28:38.839276+02:00\r\nNET  (PID:512199): Archived Log entry 132 added for B-1111517533.T-1.S-48 ID 0x7f48ce247857 LAD:1 [krse.c:4934]\r\n\r\n\r\n\r\n\r\n##############################\r\n#\r\n# Alertlog at DC1\r\n#\r\n##############################\r\n\r\n2022-08-03T23:28:38.459226+02:00\r\n rfs (PID:505753): Archived Log entry 50 added for B-1111517533.T-2.S-40 ID 0xce247857 LAD:1 [krsp.c:1256]\r\n rfs (PID:505753): No SRLs created [krsk.c:4664]\r\n2022-08-03T23:28:38.576731+02:00\r\n rfs (PID:505753): Opened log for DBID:3458508890 B-1111517533.T-2.S-41 [krsr.c:18247]\r\n2022-08-03T23:28:38.964783+02:00\r\n rfs (PID:505769): Archived Log entry 51 added for B-1111517533.T-1.S-48 ID 0xce247857 LAD:1 [krsp.c:1256]\r\n rfs (PID:505769): No SRLs created [krsk.c:4664]\r\n2022-08-03T23:28:39.026570+02:00\r\n rfs (PID:505769): Opened log for DBID:3458508890 B-1111517533.T-1.S-49 [krsr.c:18247]<\/pre>\n<p style=\"text-align: justify;\"><strong>And at the broker, we can see that everything is fine and the standby site for the pdb is orddbdc1 and the primary is the oradbdc2. And the standby is receiving and applying archivelogs correctly<\/strong>:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; show pluggable database PDBDC2 at oradbdc1;\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc1'\r\n\r\n  Data Guard Role:     Physical Standby\r\n  Con_ID:              5\r\n  Source:              con_id 4 at oradbdc2\r\n  Transport Lag:       0 seconds (computed 43 seconds ago)\r\n  Intended State:      APPLY-ON\r\n  Apply State:         Running\r\n  Apply Instance:      oradbdc11\r\n  Average Apply Rate:  1820 KByte\/s\r\n  Real Time Query:     OFF\r\n\r\nPluggable Database Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">Above you can see that everything is OK and the <em>Apply State<\/em> is in <em>Running<\/em> mode. If you saw some error like \u201c<em>ORA-16766: Redo Apply is stopped<\/em>\u201d and the <em>Apply State<\/em> in \u201c<em>Not Running<\/em>\u201d mode, please change the state to APPLY-ON:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; edit pluggable database PDBDC2 at oradbdc1 set state ='APPLY-ON';\r\nSucceeded.\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">This is even more important because after you bounce the database (planned restart or not), is possible that the apply return in a bad state. <strong>So, always remember to check the apply mode at the broker<\/strong>.<\/p>\n<h2 style=\"text-align: justify;\">Switchover PDB<\/h2>\n<p style=\"text-align: justify;\">With everything configured we can play with the switchover command. This is the reason for the DG PDB, which allows switching just one pdb while others remain in the same place. Here is my scenario I will switchover the PDBDC2 from the radbdc2 (from DC2) to oradbdc1 (DC1):<\/p>\n<p><a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-963 size-large\" src=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-1024x514.png\" alt=\"\" width=\"625\" height=\"314\" srcset=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-1024x514.png 1024w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-300x151.png 300w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-768x385.png 768w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-1536x771.png 1536w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover-624x313.png 624w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-DGPDB-Swicthover.png 1989w\" sizes=\"auto, (max-width: 625px) 100vw, 625px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify;\">&nbsp;<\/p>\n<p style=\"text-align: justify;\"><strong>Please check ISSUE #4 at the end of the post to check one detail about the switchover. <\/strong><\/p>\n<p style=\"text-align: justify;\">The command is quite simple:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; switchover to pluggable database pdbdc2 at oradbdc1;\r\nVerifying conditions for Switchover...\r\n\r\nConnected to \"oradbdc2\"\r\n  Source pluggable database is 'PDBDC2' at database 'oradbdc2'\r\n\r\nPerforming switchover NOW, please wait...\r\n\r\n  Closing pluggable database 'PDBDC2'...\r\n  Switching 'PDBDC2' to standby role...\r\nConnected to \"oradbdc1\"\r\n  Waiting for 'PDBDC2' to recover all redo data...\r\n  Stopping recovery at 'PDBDC2'...\r\n  Converting 'PDBDC2' to primary role...\r\n  Opening new primary 'PDBDC2'...\r\nConnected to \"oradbdc2\"\r\n  Waiting for redo data from new primary 'PDBDC2'...\r\n  Starting recovery at new standby 'PDBDC2'...\r\n\r\nSwitchover succeeded, new primary is \"PDBDC2\"\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">As explained in <strong>ISSUE #4<\/strong>, the command can be stuck at \u201c<em>Waiting for &#8216;PDBDC2&#8217; to recover all redo data&#8230;<\/em>\u201d. If this occurs, you need to connect to the primary database of the pdb and execute the \u201c<em>alter system archive log current\u201d<\/em> to force the generation of a new archivelog.&nbsp; This is needed because the synchronization mode is in MaxPerformance and for some reason, the process of switchover does not send the command to generate the new archive at all (current primary) instances. So, with the new archive, the standby receives and applies the last redo.<\/p>\n<p style=\"text-align: justify;\">So, after the switchover the current status is that PDBDC2 is running at DC1 as your primary site and DC2 as your standby site:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; show pluggable database PDBDC2 at oradbdc1;\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc1'\r\n\r\n  Data Guard Role:     Primary\r\n  Con_ID:              5\r\n  Active Target:       con_id 4 at oradbdc2\r\n\r\nPluggable Database Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt; show pluggable database PDBDC2 at oradbdc2;\r\nConnected to \"oradbdc2\"\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc2'\r\n\r\n  Data Guard Role:     Physical Standby\r\n  Con_ID:              4\r\n  Source:              con_id 5 at oradbdc1\r\n  Transport Lag:       0 seconds (computed 23 seconds ago)\r\n  Intended State:      APPLY-ON\r\n  Apply State:         Running\r\n  Apply Instance:      oradbdc22\r\n  Average Apply Rate:  (unknown)\r\n  Real Time Query:     OFF\r\n\r\nPluggable Database Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt;\r\nDGMGRL&gt; show configuration;\r\n\r\nConfiguration - dc1\r\n\r\n  Protection Mode: MaxPerformance\r\n  Members:\r\n  oradbdc1 - Primary database\r\n    oradbdc2 - Primary database in dc2 configuration\r\n\r\nData Guard for PDB:   Enabled in SOURCE role\r\n\r\nConfiguration Status:\r\nSUCCESS   (status updated 2 seconds ago)\r\n\r\nDGMGRL&gt; show configuration dc1\r\n\r\nConfiguration - dc1\r\n\r\n  Protection Mode: MaxPerformance\r\n  Members:\r\n  oradbdc1 - Primary database\r\n    oradbdc2 - Primary database in dc2 configuration\r\n\r\nData Guard for PDB:   Enabled in SOURCE role\r\n\r\nConfiguration Status:\r\nSUCCESS   (status updated 8 seconds ago)\r\n\r\nDGMGRL&gt; show configuration dc2\r\n\r\nConfiguration - dc2\r\n\r\nPrimary Database: oradbdc2\r\n\r\nConfiguration dc2 Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">As you can see above (compared with previous outputs) the switchover just changed the status for the PDBDC2. All the rest continue the same.<\/p>\n<p style=\"text-align: justify;\">Always remember to check the Apply State of the pdb after the switchover and even more important if you restarted some of the instances (or the entire site).<\/p>\n<h2 style=\"text-align: justify;\">DGPDB<\/h2>\n<p style=\"text-align: justify;\">The possibility protects individually your PDBS is a nice feature. As explained before it is more linked with OCI Cloud than On-Prem environments. But every client has its own configuration and different setup and this can fit for On-Prem as well.<\/p>\n<p style=\"text-align: justify;\">Just remember that (as an entire 21c) it is an innovative release and features need to be tested before you think to put it in production or not. This is the case for DGPDB, it still has some details that need to be polished (or improved) to help the configuration and also to be fully operational.<\/p>\n<p style=\"text-align: justify;\">I recommend you to read the <a href=\"https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-database\/21\/dgbkr\/examples-using-data-guard-broker-dgpdb-DGMGRL-utility.html#GUID-A4A551E3-0CB4-41DB-8CC0-FBE1E836D311\" target=\"_blank\" rel=\"noopener\">full documentation of DG for 21.7<\/a> and the <a href=\"https:\/\/blogs.oracle.com\/maa\/post\/dgpdb-available-today\" target=\"_blank\" rel=\"noopener\">post from Ludovico<\/a> (Oracle PM for DG) to understand how the DGPDB works.<\/p>\n<h2 style=\"text-align: justify;\">ISSUE #1<\/h2>\n<p style=\"text-align: justify;\">The first issue that I found was related to the remote connections between databases. If you think of normal DG situations, the dn_name for the databases is the same, so, the entries in your password files are still valid when you copy from one site to another. Here, for DG PDB, they can have completely different names and one password file will not be ready for another. I tried to use the \u201cshare\u201d password files but does not help as well.<\/p>\n<p style=\"text-align: justify;\">Below you can see that my remote connection between sites was working properly. The SYS (and even SYSDG) was working and all nodes (for all sites) can connect using the password (with tnsname and ezconnect):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus sys\/oracle@oradbdc2 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 20:12:17 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ sqlplus sys\/oracle@oradbdc1 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 20:12:21 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8n1-21c ~]$\r\n[oracle@oel8n1-21c ~]$ sqlplus sys\/oracle@\/\/oel8s-21c-scan:1521\/oradbdc2 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Tue Aug 2 19:52:47 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\n##############################\r\n#\r\n# At DC2\r\n#\r\n##############################\r\n\r\n[oracle@oel8sn1-21c ~]$ sqlplus sys\/oracle@oradbdc2 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 20:12:33 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8sn1-21c ~]$ sqlplus sys\/oracle@oradbdc1 as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Sun Jul 31 20:12:40 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt; exit\r\nDisconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n[oracle@oel8sn1-21c ~]$<\/pre>\n<p style=\"text-align: justify;\">And when I tried to add the database at dgmgrl I received the \u201c<em>ORA-1017: invalid username\/password; logon denied error<\/em>\u201d. Look below that I used the trace to support to try to troubleshoot but nothing in special was found:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; add configuration dc2 connect identifier is oradbdc2;\r\n[W000 2022-08-02T19:48:59.302+02:00] Adding a configuration [BEGIN :status := DBMS_DRS.ADD_CONFIGURATION('oradbdc2', 'dc2', 'PDB'); END;].\r\nORA-1017: invalid username\/password; logon denied\r\nDGMGRL&gt;\r\nDGMGRL&gt; add configuration connect identifier is '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel8s-21c-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oradbdc2)(SERVER=DEDICATED)))';\r\n[W000 2022-08-02T19:44:48.240+02:00] Adding a configuration [BEGIN :status := DBMS_DRS.ADD_CONFIGURATION('(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel8s-21c-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oradbdc2)(SERVER=DEDICATED)))', '', 'PDB'); END;].\r\nORA-1017: invalid username\/password; logon denied\r\nDGMGRL&gt;\r\nDGMGRL&gt; add configuration dc2 connect identifier is '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel8s-21c-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oradbdc2)(SERVER=DEDICATED)))';\r\n[W000 2022-08-02T19:49:27.689+02:00] Adding a configuration [BEGIN :status := DBMS_DRS.ADD_CONFIGURATION('(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oel8s-21c-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=oradbdc2)(SERVER=DEDICATED)))', 'dc2', 'PDB'); END;].\r\nORA-1017: invalid username\/password; logon denied\r\nDGMGRL&gt;\r\nDGMGRL&gt; add configuration dc2 connect identifier is 'oel8s-21c-scan:1521\/oradbdc2';\r\n[W000 2022-08-02T19:54:21.650+02:00] Adding a configuration [BEGIN :status := DBMS_DRS.ADD_CONFIGURATION('oel8s-21c-scan:1521\/oradbdc2', 'dc2', 'PDB'); END;].\r\nORA-1017: invalid username\/password; logon denied\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">So, maybe I made something wrong during my configuration, missed some configuration, or not understood some concept here. I tried several times: connected directly at dgmgrl (dgmgrl \/), with username password (dgmgrl sys\/oracle), with tnsnames (dgmgrl sys\/oracle@TNS), or the same with SYSDG user. Nothing worked. The only method that worked was using a wallet because it pick up the username and password from another way.<\/p>\n<h2 style=\"text-align: justify;\">ISSUE #2<\/h2>\n<p style=\"text-align: justify;\">Issue #2 is related to the ADD PLUGGABLE DATABASE that fails with the error \u201c<em>ORA-11402: Oracle Data Guard site entry not found<\/em>\u201d. Maybe this occurs because at my DC2 site I not add the configuration for DC1 too. Unfortunately, this is not clear in the documentation (if is needed to do or not). Or can be something else.<\/p>\n<p style=\"text-align: justify;\">You can see here in the file <a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/Add-Pluggable-Database-Full-Output.txt\" target=\"_blank\" rel=\"noopener\">Add-Pluggable-Database-Full-Output.txt<\/a> the output (generated with trace_level set to support) from the moment that I executed the add pluggable database command that the error appears when it connects to the target database and running the \u201c<em>create pluggable database PDBDC2 as standby from PDBDC2 at oradbdc2<\/em>\u201d.<\/p>\n<p style=\"text-align: justify;\">The simple solution was executing again the same command at dgmgrl and it completes successfully.<\/p>\n<h2 style=\"text-align: justify;\">ISSUE #3<\/h2>\n<p style=\"text-align: justify;\">Issue #3 is related to the way that the pdb is registered at the standby site. As I checked, the files (of the pdb) are not registered at the controlfile. So, the traditional way to restore the datafile (from one site to another) does not work.<\/p>\n<p style=\"text-align: justify;\">Look below that I am connected at the standby site for the pdb (oradbdc1 \u2013 DC1) and check that it appears when I execute the \u201c<em>show pdbs<\/em>\u201d, but does not appears when ai go to the cdb_datafiles or even the v$datafile:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ sqlplus \/ as sysdba\r\n\r\nSQL*Plus: Release 21.0.0.0.0 - Production on Wed Aug 3 23:16:21 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2022, Oracle.  All rights reserved.\r\n\r\n\r\nConnected to:\r\nOracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\r\nVersion 21.7.0.0.0\r\n\r\nSQL&gt;\r\nSQL&gt; show pdbs\r\n\r\n    CON_ID CON_NAME                       OPEN MODE  RESTRICTED\r\n---------- ------------------------------ ---------- ----------\r\n         2 PDB$SEED                       READ ONLY  NO\r\n         3 PDBDC1                         READ WRITE NO\r\n         5 PDBDC2                         MOUNTED\r\nSQL&gt;\r\nSQL&gt; set linesize 255\r\nSQL&gt; col FILE_NAME format a120\r\nSQL&gt; select FILE_NAME from cdb_data_files where con_id = 5;\r\n\r\nno rows selected\r\n\r\nSQL&gt;\r\nSQL&gt; select file#, NAME from v$datafile where con_id = 5;\r\n\r\nno rows selected\r\n\r\nSQL&gt;<\/pre>\n<p style=\"text-align: justify;\">So, when I try to restore from service using rman, it says that they don&#8217;t know the file:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[oracle@oel8n1-21c ~]$ rman target \/\r\n\r\nRecovery Manager: Release 21.0.0.0.0 - Production on Wed Aug 3 23:17:58 2022\r\nVersion 21.7.0.0.0\r\n\r\nCopyright (c) 1982, 2021, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nconnected to target database: ORADBDC1 (DBID=2977563181)\r\n\r\nRMAN&gt;\r\n\r\nRMAN&gt; report schema;\r\n\r\nusing target database control file instead of recovery catalog\r\nReport of database schema for database with db_unique_name ORADBDC1\r\n\r\nList of Permanent Datafiles\r\n===========================\r\nFile Size(MB) Tablespace           RB segs Datafile Name\r\n---- -------- -------------------- ------- ------------------------\r\n1    1370     SYSTEM               YES     +DATA\/ORADBDC1\/DATAFILE\/system.287.1111517335\r\n3    840      SYSAUX               NO      +DATA\/ORADBDC1\/DATAFILE\/sysaux.288.1111517379\r\n4    450      UNDOTBS1             YES     +DATA\/ORADBDC1\/DATAFILE\/undotbs1.289.1111517405\r\n5    290      PDB$SEED:SYSTEM      NO      +DATA\/ORADBDC1\/C8209F27C6B26005E053362EE80AE60E\/DATAFILE\/system.295.1111518183\r\n6    360      PDB$SEED:SYSAUX      NO      +DATA\/ORADBDC1\/C8209F27C6B26005E053362EE80AE60E\/DATAFILE\/sysaux.296.1111518183\r\n7    5        USERS                NO      +DATA\/ORADBDC1\/DATAFILE\/users.290.1111517405\r\n8    110      PDB$SEED:UNDOTBS1    NO      +DATA\/ORADBDC1\/C8209F27C6B26005E053362EE80AE60E\/DATAFILE\/undotbs1.297.1111518183\r\n9    25       UNDOTBS2             YES     +DATA\/ORADBDC1\/DATAFILE\/undotbs2.299.1111518865\r\n10   300      PDBDC1:SYSTEM        YES     +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/DATAFILE\/system.304.1111519609\r\n11   440      PDBDC1:SYSAUX        NO      +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/DATAFILE\/sysaux.305.1111519609\r\n12   110      PDBDC1:UNDOTBS1      YES     +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/DATAFILE\/undotbs1.303.1111519609\r\n13   110      PDBDC1:UNDO_2        YES     +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/DATAFILE\/undo_2.307.1111519721\r\n14   5        PDBDC1:USERS         NO      +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/DATAFILE\/users.308.1111519731\r\n\r\nList of Temporary Files\r\n=======================\r\nFile Size(MB) Tablespace           Maxsize(MB) Tempfile Name\r\n---- -------- -------------------- ----------- --------------------\r\n1    243      TEMP                 32767       +DATA\/ORADBDC1\/TEMPFILE\/temp.294.1111517479\r\n2    35       PDB$SEED:TEMP        32767       +DATA\/ORADBDC1\/E51DDAE02E247DA5E053460AA00A9806\/TEMPFILE\/temp.298.1111518203\r\n3    219      PDBDC1:TEMP          32767       +DATA\/ORADBDC1\/E51E3000E5CEBB1CE053460AA00A221D\/TEMPFILE\/temp.306.1111519663\r\n\r\nRMAN&gt;\r\n\r\nRMAN&gt; run{\r\n2&gt; set newname for pluggable database PDBDC2 to new;\r\n3&gt; restore pluggable database PDBDC2 from service oradbdc2;\r\n4&gt; }\r\n\r\nexecuting command: SET NEWNAME\r\n\r\nStarting restore at 03-AUG-22\r\nallocated channel: ORA_DISK_1\r\nchannel ORA_DISK_1: SID=315 instance=oradbdc11 device type=DISK\r\npluggable database PDBDC2 does not have any data files\r\nRMAN-00571: ===========================================================\r\nRMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============\r\nRMAN-00571: ===========================================================\r\nRMAN-03002: failure of restore command at 08\/03\/2022 23:18:44\r\nRMAN-06835: specification does not match any data file in the respository\r\n\r\nRMAN&gt;<\/pre>\n<p style=\"text-align: justify;\">And look that even the \u201c<em>report schema<\/em>\u201d does not show the datafiles. So, this can be a problem when doing the backup (or restore) of the database, if the files are unknown, they are not backed up.<\/p>\n<h2 style=\"text-align: justify;\">ISSUE #4<\/h2>\n<p style=\"text-align: justify;\">Issue #4 is one that can leave you in a bad situation. As explained during the post the standby stays waiting for the archivelog from the primary and this never arrives (since it was not informed to generate a new one after the switchover command).<\/p>\n<p style=\"text-align: justify;\">Look below that command fails with the error: \u201c<em>Target pluggable database &#8216;PDBDC2&#8217; at database &#8216;oradbdc1&#8217; has not applied all redo; cannot complete role change<\/em>\u201d:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">DGMGRL&gt; switchover to pluggable database pdbdc2 at oradbdc1;\r\nVerifying conditions for Switchover...\r\n\r\nConnected to \"oradbdc2\"\r\n  Source pluggable database is 'PDBDC2' at database 'oradbdc2'\r\n\r\nPerforming switchover NOW, please wait...\r\n\r\n  Closing pluggable database 'PDBDC2'...\r\n  Switching 'PDBDC2' to standby role...\r\nConnected to \"oradbdc1\"\r\n  Waiting for 'PDBDC2' to recover all redo data...\r\nTarget pluggable database 'PDBDC2' at database 'oradbdc1' has not applied all redo; cannot complete role change\r\n\r\nFailed.\r\nDGMGRL&gt;\r\nDGMGRL&gt; switchover to pluggable database pdbdc2 at oradbdc1;\r\nVerifying conditions for Switchover...\r\n\r\nConnected to \"oradbdc2\"\r\n  Source pluggable database is 'PDBDC2' at database 'oradbdc2'\r\n\r\n\r\n  Source pluggable database 'PDBDC2' is not open in READ WRITE mode.\r\n\r\nFailed.\r\nDGMGRL&gt;\r\nDGMGRL&gt; show pluggable database PDBDC2 at oradbdc1;\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc1'\r\n\r\n  Data Guard Role:     Physical Standby\r\n  Con_ID:              5\r\n  Source:              con_id 3 at oradbdc2\r\n  Transport Lag:       0 seconds (computed 58 seconds ago)\r\n  Intended State:      APPLY-ON\r\n  Apply State:         Running\r\n  Apply Instance:      oradbdc11\r\n  Average Apply Rate:  96 KByte\/s\r\n  Real Time Query:     OFF\r\n\r\nPluggable Database Status:\r\nSUCCESS\r\n\r\nDGMGRL&gt; show pluggable database PDBDC2 at oradbdc2;\r\nConnected to \"oradbdc2\"\r\n\r\nPluggable database 'pdbdc2' at database 'oradbdc2'\r\n\r\n  Data Guard Role:     Physical Standby\r\n  Con_ID:              3\r\n  Source:              (unknown)\r\n\r\nPluggable Database Status:\r\nORA-16911: Redo data from new source database has not been registered.\r\n\r\nDGMGRL&gt;<\/pre>\n<p style=\"text-align: justify;\">And you can see that (even worst) now both sides are defined as \u201cPhysical Standby\u201d. As pointed at my post, if your switchover is stuck at \u201c<em>Waiting for &#8216;PDBDC2&#8217; to recover all redo data<\/em>\u201d you need to connect at the current primary for database and execute \u201c<em>alter system archive log current<\/em>\u201d to generate a new archivelog, and this being transfer and applied.<\/p>\n<p style=\"text-align: justify;\">Look below the alertlog of the current standby primary of the pdb (oradbdc1 \u2013 at DC2) and check that between 22:35 and 23:42 it waits doing nothing:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">2022-08-03T23:35:14.608810+02:00\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Log +RECO\/ORADBDC1\/ARCHIVELOG\/2022_08_03\/thread_1_seq_50.501.1111793617 [krd.c:9452]\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Waiting for T-2.S-42 (in transit) [krsm.c:6205]\r\n2022-08-03T23:42:24.812496+02:00\r\nPDBDC2(5):TT04 (PID:489392): Media Recovery Log +RECO\/ORADBDC1\/ARCHIVELOG\/2022_08_03\/thread_2_seq_42.502.1111793619 [krd.c:9452]\r\nPDBDC2(5):End of lifespan for PDB at SCN 0x0000000000b429d7 due to SWITCHOVER\r\n2022-08-03T23:42:25.278394+02:00\r\nPDBDC2(5):Incomplete recovery applied all redo ever generated.\r\nPDBDC2(5):Recovery completed through change 11807191 time 08\/03\/2022 23:35:13\r\n2022-08-03T23:42:25.952898+02:00\r\nALTER PLUGGABLE DATABASE PDBDC2 RECOVER MANAGED STANDBY DATABASE CANCEL\r\n2022-08-03T23:42:25.962198+02:00\r\n.... (PID:493823): Managed Recovery Canceled [dbsdrv.c:16992]\r\nCompleted: ALTER PLUGGABLE DATABASE PDBDC2 RECOVER MANAGED STANDBY DATABASE CANCEL\r\nALTER PLUGGABLE DATABASE PDBDC2 COMMIT TO SWITCHOVER\r\nPDBDC2(5):Pluggable database PDBDC2 pseudo opening in read only\r\n\u2026<\/pre>\n<p style=\"text-align: justify;\">And the interesting is that if you connect at the current primary it is in a close state. You can see at alertlog that pdb was closed but no archivelog was generated after that. So, the \u201cclose\u201d and an archivelog with the \u201cend of lifespan\u201d for pdb never arrived:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">2022-08-03T23:35:04.867884+02:00\r\nPDBDC2(4):Pluggable database PDBDC2 closing\r\nPDBDC2(4):Increasing priority of 1 RS\r\nPDBDC2(4):JIT: pid 438699 requesting stop\r\nPDBDC2(4):Closing sequence subsystem (48992788888).\r\nPDBDC2(4):Buffer Cache flush started: 4\r\nPDBDC2(4):Buffer Cache flush finished: 4\r\nPDBDC2(4):queued detach DA request 0x8fd84e00 for pdb 4, ospid 438699\r\n2022-08-03T23:35:05.480217+02:00\r\nDomain Action Reconfiguration started (domid 4, new da inc 5, da windows 1, cluster inc 4)\r\nInstance 2 is detaching from domain 4 (lazy abort? 0, recovery member? 0)\r\n Global Resource Directory partially frozen for domain action\r\n* domain detach - domain 4 valid ? 1\r\n Non-local Process blocks cleaned out\r\n Set master node info\r\n Dwn-cvts replayed, VALBLKs dubious\r\n All grantable enqueues granted\r\nfreeing the fusion rht of pdb 4\r\nfreeing the pdb enqueue rht\r\nDomain Action Reconfiguration complete (total time 0.1 secs)\r\n2022-08-03T23:35:05.599572+02:00\r\nPluggable database PDBDC2 closed\r\nDecreasing priority of 1 RS\r\n2022-08-03T23:35:06.013872+02:00\r\nfreeing rdom 4\r\n2022-08-03T23:35:09.896222+02:00\r\nSkipping local setting of Resource Manager plan DEFAULT_MAINTENANCE_PLAN via scheduler window [52c0] because plan is already set.\r\n2022-08-03T23:41:16.511405+02:00\r\nControl autobackup written to DISK device\r\n\r\nhandle '+RECO\/ORADBDC2\/AUTOBACKUP\/2022_08_03\/s_1111794075.438.1111794077'\r\n\r\n2022-08-03T23:42:24.538274+02:00\r\nThread 2 advanced to log sequence 43 (LGWR switch),  current SCN: 11814659\r\n  Current log# 3 seq# 43 mem# 0: +DATA\/ORADBDC2\/ONLINELOG\/group_3.297.1111518715\r\n  Current log# 3 seq# 43 mem# 1: +RECO\/ORADBDC2\/ONLINELOG\/group_3.310.1111518721\r\n2022-08-03T23:42:24.699608+02:00<\/pre>\n<p style=\"text-align: justify;\">The full outputs from the alertlogs can be seen <a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/Alertlogs-From-Databases.txt\" target=\"_blank\" rel=\"noopener\">here in this file<\/a> if you want to check the whole process.<\/p>\n<p style=\"text-align: justify;\">&nbsp;<\/p>\n<p style=\"text-align: justify;\"><strong>Disclaimer<\/strong>: \u201c<em>The postings on this site are my own and don\u2019t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.<\/em>\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Since the 21c was public available the Data Guard per Pluggable Database &#8211; DG PDB &#8211; was intended to be there, but Oracle needed more time to make things work and some weeks ago released the feature with the 21.7 version. Here in this post, I will show to configure it and also how to [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[164,50,41,29,5],"tags":[166,67,66,170,65,171],"class_list":["post-950","post","type-post","status-publish","format-standard","hentry","category-21c","category-broker","category-data-guard","category-database","category-oracle","tag-21c","tag-data-guard","tag-database","tag-dg-pdb","tag-oracle","tag-troubleshoot"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>21c, DG PDB - Fernando Simon<\/title>\n<meta name=\"description\" content=\"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"21c, DG PDB - Fernando Simon\" \/>\n<meta property=\"og:description\" content=\"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\" \/>\n<meta property=\"og:site_name\" content=\"Fernando Simon\" \/>\n<meta property=\"article:published_time\" content=\"2022-08-07T21:14:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-25T22:58:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png\" \/>\n<meta name=\"author\" content=\"Simon\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Simon\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"16 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\"},\"author\":{\"name\":\"Simon\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"headline\":\"21c, DG PDB\",\"datePublished\":\"2022-08-07T21:14:30+00:00\",\"dateModified\":\"2023-12-25T22:58:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\"},\"wordCount\":3429,\"commentCount\":1,\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png\",\"keywords\":[\"21c\",\"Data Guard\",\"Database\",\"DG PDB\",\"Oracle\",\"Troubleshoot\"],\"articleSection\":[\"21c\",\"Broker\",\"Data Guard\",\"Database\",\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\",\"name\":\"21c, DG PDB - Fernando Simon\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png\",\"datePublished\":\"2022-08-07T21:14:30+00:00\",\"dateModified\":\"2023-12-25T22:58:55+00:00\",\"author\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"description\":\"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png\",\"contentUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png\",\"width\":1988,\"height\":932},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fernandosimon.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"21c, DG PDB\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/\",\"name\":\"Fernando Simon\",\"description\":\"Have you hugged your backup today?\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\",\"name\":\"Simon\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"caption\":\"Simon\"},\"sameAs\":[\"http:\/\/www.fernandosimon.com\"],\"url\":\"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"21c, DG PDB - Fernando Simon","description":"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/","og_locale":"en_US","og_type":"article","og_title":"21c, DG PDB - Fernando Simon","og_description":"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.","og_url":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/","og_site_name":"Fernando Simon","article_published_time":"2022-08-07T21:14:30+00:00","article_modified_time":"2023-12-25T22:58:55+00:00","og_image":[{"url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png","type":"","width":"","height":""}],"author":"Simon","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Simon","Est. reading time":"16 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#article","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/"},"author":{"name":"Simon","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"headline":"21c, DG PDB","datePublished":"2022-08-07T21:14:30+00:00","dateModified":"2023-12-25T22:58:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/"},"wordCount":3429,"commentCount":1,"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png","keywords":["21c","Data Guard","Database","DG PDB","Oracle","Troubleshoot"],"articleSection":["21c","Broker","Data Guard","Database","Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/","url":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/","name":"21c, DG PDB - Fernando Simon","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage"},"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch-1024x480.png","datePublished":"2022-08-07T21:14:30+00:00","dateModified":"2023-12-25T22:58:55+00:00","author":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"description":"Check how to configure the new feature DG PDB for 21c. All the steps, logs, outputs, errors, pitfalls, and troubleshoot are described. Environment with Data Guard, RAC, and ASM.","breadcrumb":{"@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#primaryimage","url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png","contentUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2022\/08\/DGDPDB-Arch.png","width":1988,"height":932},{"@type":"BreadcrumbList","@id":"https:\/\/www.fernandosimon.com\/blog\/21c-dg-pdb\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fernandosimon.com\/blog\/"},{"@type":"ListItem","position":2,"name":"21c, DG PDB"}]},{"@type":"WebSite","@id":"https:\/\/www.fernandosimon.com\/blog\/#website","url":"https:\/\/www.fernandosimon.com\/blog\/","name":"Fernando Simon","description":"Have you hugged your backup today?","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9","name":"Simon","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","caption":"Simon"},"sameAs":["http:\/\/www.fernandosimon.com"],"url":"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5ofTp-fk","_links":{"self":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/950","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/comments?post=950"}],"version-history":[{"count":0,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/950\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/media?parent=950"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/categories?post=950"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/tags?post=950"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}