{"id":461,"date":"2019-03-18T19:46:53","date_gmt":"2019-03-18T22:46:53","guid":{"rendered":"http:\/\/www.fernandosimon.com\/blog\/?p=461"},"modified":"2020-06-11T13:22:48","modified_gmt":"2020-06-11T16:22:48","slug":"observer-where","status":"publish","type":"post","link":"https:\/\/www.fernandosimon.com\/blog\/observer-where\/","title":{"rendered":"Observer, Where?"},"content":{"rendered":"\r\n

Some months ago I got one error with Oracle Data Guard and now I had time to review it again and write this article. Just to be clear since the beginning, the discussion here is not about the error itself, but about the circumstances that generated it.<\/p>\r\n

The environment described here follows, at least, the most common best practices for DG by Oracle. Have 1 dedicated server for each one: Primary Database, Physical Standby Database, and Observer. The primary and standby reside in different data centers in different cities, dedicated network for interconnecting between sites, protection mode was Maximum Availability, and runs with Fast-Start Failover enabled (with 30 seconds for threshold). The version here is 12.2 but will be the same for 19c. So, nothing so bad in the environment, basic DG configuration trying to follow the best practices.<\/p>\r\n

<\/p>\r\n

\"\"<\/a>But, one day, application servers running, primary Linux DB server running, but the database itself down. Looking for the cause, found in the broker log (simulated in my test environment):<\/p>\r\n

03\/10\/2019 12:48:05\r\nLGWR: FSFO SetState(st=2 \"UNSYNC\", fl=0x2 \"WAIT\", ob=0x0, tgt=0, v=0)\r\nLGWR: FSFO SetState(\"UNSYNC\", 0x2) operation requires an ack\r\n        Primary database will shutdown within 30 seconds if permission\r\n         is not granted from Observer or FSFO target standby to proceed\r\nLGWR:   current in-memory FSFO state flags=0x40001, version=46\r\n03\/10\/2019 12:48:19\r\nA Fast-Start Failover target switch is necessary because the primary cannot reach the Fast-Start Failover target standby database\r\nA target switch was not attempted because the observer has not pinging primary recently.\r\nFSFP network call timeout. Killing process FSFP.\r\n03\/10\/2019 12:48:36\r\nNotifying Oracle Clusterware to disable services and monitoring because primary will be shutdown\r\nPrimary has heard from neither observer nor target standby\r\n        within FastStartFailoverThreshold seconds. It is\r\n        likely an automatic failover has already occurred.\r\n        The primary is shutting down.\r\nLGWR: FSFO SetState(st=8 \"FO PENDING\", fl=0x0 \"\", ob=0x0, tgt=0, v=0)\r\nLGWR: Shutdown primary instance 1 now because the primary has been isolated\r\n<\/pre>\r\n
And in the alertlog:<\/p>\r\n
2019-03-10T12:48:35.860142+01:00\r\nStarting background process FSFP\r\n2019-03-10T12:48:35.870802+01:00\r\nFSFP started with pid=30, OS id=6055\r\n2019-03-10T12:48:36.875828+01:00\r\nPrimary has heard from neither observer nor target standby within FastStartFailoverThreshold seconds.\r\nIt is likely an automatic failover has already occurred. Primary is shutting down.\r\n2019-03-10T12:48:36.876434+01:00\r\nErrors in file \/u01\/app\/oracle\/diag\/rdbms\/orcl\/orcl\/trace\/orcl_lgwr_2472.trc:\r\nORA-16830: primary isolated from fast-start failover partners longer than FastStartFailoverThreshold seconds: shutting down\r\nLGWR (ospid: 2472): terminating the instance due to error 16830\r\n<\/pre>\r\n
But this is (and was) not a DG problem, the DG made what was design to do. Primary lost the communication with the Standby and Observer and after the Fast-Start Failover threshold, FSFP killed the primary because it doesn\u2019t know if it was evicted and want to avoid split-brain (or something similar). Worked as designed!<\/p>\r\n
As I wrote before the main question here is not the ORA-XXXX error, but the circumstances. In this case, by design definition (and probably based in the docs), chosen to put the observer in the same site than standby. But, because one failure in the standby datacenter (just in the enclosure that runs blades for Oracle stuffs, the application continued to runs), the entire database was unavailable. One outage in standby datacenter, shut down the primary database even DG running in Maximum Availability mode.<\/p>\r\n
\"\"<\/a><\/p>\r\n
As you can imagine, because of the design decision \u201cwhere to put the observer\u201d, everything was down. If the observer was running in the primary datacenter, nothing supposed to occurs. But here it is the point for this post: \u201cWhere you put the Observer?<\/strong><\/em>\u201d, \u201cPrimary site? Standby site?<\/em><\/strong>\u201d, and \u201cWhy<\/strong>?\u201d, \u201cHow you based this decision?<\/strong>\u201d. It appears to be a simple question to answer, but there are a lot of pros and cons, and there is not much information about that.<\/p>\r\n
If you search in the Oracle docs they recommend to put the observer in a third datacenter, isolated from the others, or in the same network than application, or in the standby datacenter. Here: https:\/\/www.oracle.com\/technetwork\/database\/availability\/maa-roletransitionbp-2621582.pdf<\/a> (page 9). But, where are the pros and cons of the decision? And how many clients have a third datacenter? And if you search about were put the observer over the google, the 99% spread the same information (that go in the opposite of docs) \u201cprimary site\u201d. But again, \u201cWhy?\u201d.<\/p>\r\n
\"\"<\/a>The observer in primary site:<\/p>\r\n