2019-05-05 20:13:44.308 FSFP: Deleting connection to site 2 2019-05-05 20:13:44.308 FSFP: Network I/O to database orcls failed, status = ORA-12609. 2019-05-05 20:13:59.309 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:13:59.309 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:13:59.309 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:13:59.309 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:14:06.502 LGWR: FSFO SetState(st=2 "UNSYNC", fl=0x2 "WAIT", ob=0x0, tgt=0, v=0) 2019-05-05 20:14:06.503 LGWR: FSFO SetState("UNSYNC", 0x2) operation requires an ack 2019-05-05 20:14:06.504 LGWR: Primary database will shutdown within 240 seconds if permission 2019-05-05 20:14:06.504 is not granted from Observer or FSFO target standby to proceed 2019-05-05 20:14:06.505 LGWR: current in-memory FSFO state flags=0x40801, version=128 2019-05-05 20:14:10.935 DMON: A Fast-Start Failover target switch is necessary because the primary cannot reach the Fast-Start Failover target standby database 2019-05-05 20:14:10.935 DMON: A target switch was not attempted because the observer has not pinging primary recently. 2019-05-05 20:14:21.506 LGWR: still awaiting FSFO ack after 15 seconds 2019-05-05 20:14:26.936 DMON: FSFP network call timeout. Killing process FSFP. 2019-05-05 20:14:27.013 RSM0: HEALTH CHECK WARNING: ORA-16735: primary redo generation suspended 2019-05-05 20:14:27.018 RSM0: HEALTH CHECK ERROR: ORA-16820: fast-start failover observer is no longer observing this database 2019-05-05 20:14:27.018 00001000 177045427 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16825 2019-05-05 20:14:27.019 00001000 177045427 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16825 2019-05-05 20:14:36.506 LGWR: still awaiting FSFO ack after 30 seconds 2019-05-05 20:14:41.968 DMON: Creating process FSFP 2019-05-05 20:14:44.976 FSFP: Process started 2019-05-05 20:14:44.976 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:14:44.976 FSFP: Preparing to switch master observer from observer 1 to observer 2 2019-05-05 20:14:44.976 FSFP: FSFO SetState(st=43 "SET SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:14:44.976 DMON: FSFP successfully started 2019-05-05 20:14:44.977 FSFP: persisting FSFO state flags=0x10040001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:14:44.979 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:14:44.980 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:14:44.980 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:14:44.981 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:14:44.981 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:14:51.506 LGWR: still awaiting FSFO ack after 45 seconds 2019-05-05 20:14:55.120 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:14:55.120 FSFP: FSFO SetState(st=44 "CLR SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:14:55.120 FSFP: persisting FSFO state flags=0x40001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:15:06.507 LGWR: still awaiting FSFO ack after 60 seconds 2019-05-05 20:15:10.124 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:15:10.124 FSFP: Preparing to switch master observer from observer 1 to observer 2 2019-05-05 20:15:10.124 FSFP: FSFO SetState(st=43 "SET SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:15:10.124 FSFP: persisting FSFO state flags=0x10040001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:15:10.128 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:15:10.128 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:15:10.128 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:15:10.128 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:15:10.128 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:15:14.979 DMON: A Fast-Start Failover target switch is necessary because the primary cannot reach the Fast-Start Failover target standby database 2019-05-05 20:15:14.980 DMON: A target switch was not attempted because the observer has not pinging primary recently. 2019-05-05 20:15:21.246 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:15:21.247 FSFP: FSFO SetState(st=44 "CLR SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:15:21.247 FSFP: persisting FSFO state flags=0x40001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:15:21.507 LGWR: still awaiting FSFO ack after 75 seconds 2019-05-05 20:15:36.250 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:15:36.250 FSFP: Switching master observer cannot start since there is no healthy backup observer 2019-05-05 20:15:36.250 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:15:36.250 FSFP: Data Guard Broker cannot complete master observer switch since no healthy backup observer 2019-05-05 20:15:36.250 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:15:36.250 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:15:36.250 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:15:36.250 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:15:36.507 LGWR: still awaiting FSFO ack after 90 seconds 2019-05-05 20:15:47.376 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:15:51.508 LGWR: still awaiting FSFO ack after 105 seconds 2019-05-05 20:16:02.377 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:16:02.377 FSFP: Preparing to switch master observer from observer 1 to observer 2 2019-05-05 20:16:02.377 FSFP: FSFO SetState(st=43 "SET SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:16:02.377 FSFP: persisting FSFO state flags=0x10040001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:16:02.380 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:16:02.380 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:16:02.380 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:16:02.380 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:16:02.380 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:16:06.508 LGWR: still awaiting FSFO ack after 120 seconds 2019-05-05 20:16:13.502 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:16:13.502 FSFP: FSFO SetState(st=44 "CLR SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:16:13.502 FSFP: persisting FSFO state flags=0x40001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:16:14.982 DMON: A Fast-Start Failover target switch is necessary because the primary cannot reach the Fast-Start Failover target standby database 2019-05-05 20:16:14.983 DMON: A target switch was not attempted because the observer has not pinging primary recently. 2019-05-05 20:16:21.507 LGWR: still awaiting FSFO ack after 135 seconds 2019-05-05 20:16:28.505 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:16:28.505 FSFP: Preparing to switch master observer from observer 1 to observer 2 2019-05-05 20:16:28.505 FSFP: FSFO SetState(st=43 "SET SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:16:28.505 FSFP: persisting FSFO state flags=0x10040001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:16:28.507 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:16:28.507 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:16:28.508 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:16:28.508 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:16:28.508 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:16:36.509 LGWR: still awaiting FSFO ack after 150 seconds 2019-05-05 20:16:39.493 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:16:39.494 FSFP: FSFO SetState(st=44 "CLR SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:16:39.494 FSFP: persisting FSFO state flags=0x40001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:16:51.509 LGWR: still awaiting FSFO ack after 165 seconds 2019-05-05 20:16:54.498 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:16:54.498 FSFP: Switching master observer cannot start since there is no healthy backup observer 2019-05-05 20:16:54.498 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:16:54.498 FSFP: Data Guard Broker cannot complete master observer switch since no healthy backup observer 2019-05-05 20:16:54.498 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:16:54.498 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:16:54.498 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:16:54.498 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:17:05.760 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:17:06.508 LGWR: still awaiting FSFO ack after 180 seconds 2019-05-05 20:17:14.986 DMON: A Fast-Start Failover target switch is necessary because the primary cannot reach the Fast-Start Failover target standby database 2019-05-05 20:17:14.986 DMON: A target switch was not attempted because the observer has not pinging primary recently. 2019-05-05 20:17:20.761 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:17:20.761 FSFP: Preparing to switch master observer from observer 1 to observer 2 2019-05-05 20:17:20.761 FSFP: FSFO SetState(st=43 "SET SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:17:20.761 FSFP: persisting FSFO state flags=0x10040001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:17:20.764 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:17:20.765 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:17:20.765 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:17:20.766 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:17:20.766 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:17:21.510 LGWR: still awaiting FSFO ack after 195 seconds 2019-05-05 20:17:31.886 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:17:31.886 FSFP: FSFO SetState(st=44 "CLR SWOB INPRG", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:17:31.886 FSFP: persisting FSFO state flags=0x40001, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:17:36.511 LGWR: still awaiting FSFO ack after 210 seconds 2019-05-05 20:17:44.986 00001000 177045427 DMON: Data Guard Broker terminating NSV1, timed out waiting for a response from database orcls 2019-05-05 20:17:45.987 DMON: NSV1 has been killed 2019-05-05 20:17:45.987 00001000 177045427 DMON: Database orcls returned ORA-16662 2019-05-05 20:17:45.987 00001000 177045427 for opcode = HEALTH_CHECK, phase = BEGIN, req_id = 1.1.177045427 2019-05-05 20:17:45.987 DMON: Data Guard Broker Status Summary: 2019-05-05 20:17:45.987 DMON: Type Name Severity Status 2019-05-05 20:17:45.987 DMON: Configuration dgconfig Warning ORA-16607 2019-05-05 20:17:45.987 DMON: Primary Database orcl Error ORA-16825 2019-05-05 20:17:45.987 DMON: Physical Standby Database orcls Error ORA-16662 2019-05-05 20:17:46.889 FSFP: The ping from current master observer (No. 1) to primary database timed out. 2019-05-05 20:17:46.889 FSFP: Switching master observer cannot start since there is no healthy backup observer 2019-05-05 20:17:46.889 FSFP: Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database 2019-05-05 20:17:46.890 FSFP: Data Guard Broker cannot complete master observer switch since no healthy backup observer 2019-05-05 20:17:46.890 FSFP: NET Alias translated: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dbstb_dg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcls) (UR=A))) 2019-05-05 20:17:46.890 FSFP: Net link using RFIUPI_DB_CDESC, site=2, inst=0 2019-05-05 20:17:46.890 FSFP: Connect to member orcls using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbstb_dg)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(UR=A)(SERVICE_NAME=orcls_DGB))) 2019-05-05 20:17:46.890 FSFP: 0 x 10 seconds network retry READY 2019-05-05 20:17:51.510 LGWR: still awaiting FSFO ack after 225 seconds 2019-05-05 20:17:58.357 FSFP: Failed to connect to remote database orcls. Error is ORA-12543 2019-05-05 20:18:06.510 LGWR: still awaiting FSFO ack after 240 seconds 2019-05-05 20:18:08.511 LGWR: Notifying Oracle Clusterware to disable services and monitoring because primary will be shutdown 2019-05-05 20:18:08.511 LGWR: Primary has heard from neither observer nor target standby 2019-05-05 20:18:08.511 LGWR: within FastStartFailoverThreshold seconds. It is 2019-05-05 20:18:08.511 LGWR: likely an automatic failover has already occurred. 2019-05-05 20:18:08.511 LGWR: The primary is shutting down. 2019-05-05 20:18:08.511 LGWR: Becoming the shutdown master 2019-05-05 20:18:08.511 LGWR: FSFO SetState(st=8 "FO PENDING", fl=0x0 "", ob=0x0, tgt=0, v=0) 2019-05-05 20:18:08.511 LGWR: persisting FSFO state flags=0x40021, version=128, enver=123,target=2, lad=, obid=0x32cc2adc (852241116), threshold=240, laglim=30, obslim=30 2019-05-05 20:18:08.514 CLSR: clsr_crs_status CRS not configured, config = 2 . 2019-05-05 20:18:08.514 LGWR: Shutdown primary instance 1 now because the primary has been isolated