###################################################### ALERTLOG FROM DBA19 ###################################################### 2020-03-22T17:18:40.594647+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_ckpt_952.trc: 2020-03-22T17:18:40.642466+01:00 WARNING: Write Failed. group:1 disk:1 AU:575 offset:49152 size:16384 path:ORCL:CELLI01 incarnation:0xf0f70497 asynchronous result:'I/O error' subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f53516e29d8 bufp:0x7f5351683000 osderr1:0x3 osderr2:0x2e IO elapsed time: 0 usec Time waited on I/O: 0 usec 2020-03-22T17:18:40.652638+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_ckpt_952.trc: ORA-15080: synchronous I/O operation failed to write block 3 of disk 1 in disk group DATA ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group 1 on disk 1 allocation unit 575 NOTE: process _ckpt_dba19 (952) initiating offline of disk 1.4042720407 (CELLI01) with mask 0x7e in group 1 (DATA) with client assisting WARNING: failed to offline some disks corresponding to the mirrors of virtual extent 0 of file 260 in group 1 WARNING: group 1 file 260 vxn 0 block 3 write I/O failed 2020-03-22T17:18:40.717546+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_ckpt_952.trc: ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBA19/CONTROLFILE/current.260.1035154045' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] 2020-03-22T17:18:40.722912+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_ckpt_952.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBA19/CONTROLFILE/current.260.1035154045' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_ckpt_952.trc (incident=50712): ORA-221 [] [] [] [] [] [] [] [] [] [] [] [] Incident details in: /u01/app/oracle/diag/rdbms/dba19/DBA19/incident/incdir_50712/DBA19_ckpt_952_i50712.trc 2020-03-22T17:18:40.801137+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: 2020-03-22T17:18:40.801223+01:00 WARNING: Read Failed. group:1 disk:1 AU:575 offset:16384 size:16384 path:ORCL:CELLI01 incarnation:0xf0f70497 synchronous result:'I/O error' subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f5fb7b03778 bufp:0x7f5fb268d000 osderr1:0x3 osderr2:0x2e (mirrored read) IO elapsed time: 0 usec Time waited on I/O: 0 usec WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group [1.3028300217] from disk CELLI01 allocation unit 575 reason error; if possible, will try another mirror side WARNING: group 1 file 260 vxn 0 block 1 read I/O failed 2020-03-22T17:18:40.804638+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: NOTE: Suppressing further IO Read errors on group:1 disk:1 WARNING: Read Failed. group:1 disk:1 AU:575 offset:16384 size:16384 path:ORCL:CELLI01 incarnation:0xf0f70497 synchronous result:'I/O error' subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f5fb7b03778 bufp:0x7f5fb268d000 osderr1:0x3 osderr2:0x2e (mirrored read) IO elapsed time: 0 usec Time waited on I/O: 0 usec WARNING: group 1 file 260 vxn 0 block 1 read I/O failed 2020-03-22T17:18:40.813121+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: WARNING: group 1 file 260 vxn 0 block 1 read I/O failed 2020-03-22T17:18:40.817052+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: 2020-03-22T17:18:40.826079+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] ERROR: cannot read disk header of disk CELLI01 (1:4042720407) NOTE: process _mz00_dba19 (12066) initiating offline of disk 1.4042720407 (CELLI01) with mask 0x7e in group 1 (DATA) with client assisting 2020-03-22T17:18:40.885491+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] 2020-03-22T17:18:41.145698+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: ORA-00312: online log 1 thread 1: '+DATA/DBA19/ONLINELOG/group_1.263.1035154063' ORA-17503: ksfdopn:2 Failed to open file +DATA/DBA19/ONLINELOG/group_1.263.1035154063 ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] 2020-03-22T17:18:41.243972+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: ORA-00313: open failed for members of log group 1 of thread 1 ORA-00312: online log 1 thread 1: '+DATA/DBA19/ONLINELOG/group_1.263.1035154063' ORA-17503: ksfdopn:2 Failed to open file +DATA/DBA19/ONLINELOG/group_1.263.1035154063 ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] 2020-03-22T17:18:41.330803+01:00 Errors in file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_mz00_12066.trc: ORA-00313: open failed for members of log group 1 of thread 1 ORA-00312: online log 1 thread 1: '+DATA/DBA19/ONLINELOG/group_1.263.1035154063' ORA-17503: ksfdopn:2 Failed to open file +DATA/DBA19/ONLINELOG/group_1.263.1035154063 ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "CELLI01" in group "DATA" may result in a data loss ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error = [3], mesg = [I/O Error] 2020-03-22T17:18:41.631424+01:00 USER (ospid: 952): terminating the instance due to ORA error 221 2020-03-22T17:18:41.723065+01:00 System state dump requested by (instance=1, osid=952 (CKPT)), summary=[abnormal instance termination]. error - 'Instance is terminating. ' System State dumped to trace file /u01/app/oracle/diag/rdbms/dba19/DBA19/trace/DBA19_diag_920.trc 2020-03-22T17:18:41.843516+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:42.325939+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:42.965813+01:00 Dumping diagnostic data in directory=[cdmp_20200322171841], requested by (instance=1, osid=952 (CKPT)), summary=[abnormal instance termination]. 2020-03-22T17:18:43.380410+01:00 License high water mark = 10 2020-03-22T17:18:43.869527+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:44.171040+01:00 Instance terminated by USER, pid = 952 2020-03-22T17:18:44.200686+01:00 Warning: 3 processes are still attacheded to shmid 917508: (size: 36864 bytes, creator pid: 738, last attach/detach pid: 12109) 2020-03-22T17:18:44.381210+01:00 USER(prelim) (ospid: 12094): terminating the instance 2020-03-22T17:18:44.382164+01:00 Instance terminated by USER(prelim), pid = 12094 ###################################################### ALERTLOG FROM DBB19 ###################################################### 2020-03-22T17:18:45.703948+01:00 Starting background process AMB1 2020-03-22T17:18:45.718218+01:00 AMB1 started with pid=80, OS id=12119 *********************************************************************** Fatal NI connect error 12514, connecting to: (DESCRIPTION=(TCP_USER_TIMEOUT=1)(RECV_TIMEOUT=10)(CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.2.5)(PORT=1525)))(CONNECT_DATA=(SERVICE_NAME=+ASM_DATA)(CID=(PROGRAM=oracle)(HOST=asmrec.oralocal)(USER=oracle)))) VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production Version 19.6.0.0.0 Time: 22-MAR-2020 17:18:46 Tracing not turned on. Tns error struct: ns main err code: 12564 TNS-12564: TNS:connection refused ns secondary err code: 12560 nt main err code: 524 TNS-00524: Current operation is still in progress nt secondary err code: 115 nt OS err code: 0 2020-03-22T17:18:47.237559+01:00 *********************************************************************** Fatal NI connect error 12514, connecting to: (DESCRIPTION=(TCP_USER_TIMEOUT=1)(RECV_TIMEOUT=10)(CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.2.5)(PORT=1525)))(CONNECT_DATA=(SERVICE_NAME=+ASM_DATA)(CID=(PROGRAM=oracle)(HOST=asmrec.oralocal)(USER=oracle)))) VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production Version 19.6.0.0.0 Time: 22-MAR-2020 17:18:47 Tracing not turned on. Tns error struct: ns main err code: 12564 TNS-12564: TNS:connection refused ns secondary err code: 12560 nt main err code: 524 TNS-00524: Current operation is still in progress nt secondary err code: 115 nt OS err code: 0 2020-03-22T17:18:47.246789+01:00 WARNING: failed to register AMB1 with ASM instance 2020-03-22T17:18:47.247227+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbb19/DBB19/trace/DBB19_amb1_12119.trc: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor ORA-12514: TNS:listener does not currently know of service requested in connect descriptor WARNING: AMB1 exiting with error 2020-03-22T17:18:47.412366+01:00 NOTE: ASMB (13198) dismounting group 1 WARNING: group 1 is being dismounted. WARNING: ASMB force dismounting group 1 (DATA) due to failover SUCCESS: diskgroup DATA was dismounted NOTE: ASMB (13198) dismounted 1 group(s) 2020-03-22T17:18:48.266464+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbb19/DBB19/trace/DBB19_ckpt_13176.trc: ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBB19/CONTROLFILE/current.272.1035154347' ORA-15078: ASM diskgroup was forcibly dismounted ORA-15078: ASM diskgroup was forcibly dismounted 2020-03-22T17:18:48.269813+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbb19/DBB19/trace/DBB19_ckpt_13176.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBB19/CONTROLFILE/current.272.1035154347' ORA-15078: ASM diskgroup was forcibly dismounted ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u01/app/oracle/diag/rdbms/dbb19/DBB19/trace/DBB19_ckpt_13176.trc (incident=48287): ORA-221 [] [] [] [] [] [] [] [] [] [] [] [] Incident details in: /u01/app/oracle/diag/rdbms/dbb19/DBB19/incident/incdir_48287/DBB19_ckpt_13176_i48287.trc USER (ospid: 13176): terminating the instance due to ORA error 221 2020-03-22T17:18:49.124305+01:00 System state dump requested by (instance=1, osid=13176 (CKPT)), summary=[abnormal instance termination]. error - 'Instance is terminating. ' System State dumped to trace file /u01/app/oracle/diag/rdbms/dbb19/DBB19/trace/DBB19_diag_13145.trc 2020-03-22T17:18:50.330652+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:50.658097+01:00 Dumping diagnostic data in directory=[cdmp_20200322171849], requested by (instance=1, osid=13176 (CKPT)), summary=[abnormal instance termination]. 2020-03-22T17:18:51.386421+01:00 License high water mark = 9 2020-03-22T17:18:51.824760+01:00 Instance terminated by USER, pid = 13176 2020-03-22T17:18:51.865742+01:00 Warning: 2 processes are still attacheded to shmid 1048584: (size: 36864 bytes, creator pid: 13083, last attach/detach pid: 13130) 2020-03-22T17:18:52.387119+01:00 USER(prelim) (ospid: 12180): terminating the instance 2020-03-22T17:18:52.387849+01:00 Instance terminated by USER(prelim), pid = 12180 ###################################################### ALERTLOG FROM DBC19 ###################################################### 2020-03-22T17:18:46.500063+01:00 Starting background process AMB1 2020-03-22T17:18:46.522412+01:00 AMB1 started with pid=78, OS id=12124 *********************************************************************** Fatal NI connect error 12514, connecting to: (DESCRIPTION=(TCP_USER_TIMEOUT=1)(RECV_TIMEOUT=10)(CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.2.5)(PORT=1525)))(CONNECT_DATA=(SERVICE_NAME=+ASM_DATA)(CID=(PROGRAM=oracle)(HOST=asmrec.oralocal)(USER=oracle)))) VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production Version 19.6.0.0.0 Time: 22-MAR-2020 17:18:46 Tracing not turned on. Tns error struct: ns main err code: 12564 TNS-12564: TNS:connection refused ns secondary err code: 12560 nt main err code: 524 TNS-00524: Current operation is still in progress nt secondary err code: 115 nt OS err code: 0 2020-03-22T17:18:47.959399+01:00 *********************************************************************** Fatal NI connect error 12514, connecting to: (DESCRIPTION=(TCP_USER_TIMEOUT=1)(RECV_TIMEOUT=10)(CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.2.5)(PORT=1525)))(CONNECT_DATA=(SERVICE_NAME=+ASM_DATA)(CID=(PROGRAM=oracle)(HOST=asmrec.oralocal)(USER=oracle)))) VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production Version 19.6.0.0.0 Time: 22-MAR-2020 17:18:47 Tracing not turned on. Tns error struct: ns main err code: 12564 TNS-12564: TNS:connection refused ns secondary err code: 12560 nt main err code: 524 TNS-00524: Current operation is still in progress nt secondary err code: 115 nt OS err code: 0 2020-03-22T17:18:47.962934+01:00 WARNING: failed to register AMB1 with ASM instance 2020-03-22T17:18:47.963033+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_amb1_12124.trc: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor ORA-12514: TNS:listener does not currently know of service requested in connect descriptor WARNING: AMB1 exiting with error 2020-03-22T17:18:48.002819+01:00 NOTE: ASMB (16238) dismounting group 1 WARNING: group 1 is being dismounted. WARNING: ASMB force dismounting group 1 (DATA) due to failover SUCCESS: diskgroup DATA was dismounted 2020-03-22T17:18:49.038895+01:00 NOTE: ASMB (16238) dismounted 1 group(s) 2020-03-22T17:18:49.244809+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_ckpt_16216.trc: ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBC19/CONTROLFILE/current.273.1035154345' ORA-15078: ASM diskgroup was forcibly dismounted ORA-15078: ASM diskgroup was forcibly dismounted 2020-03-22T17:18:49.248412+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_ckpt_16216.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '+DATA/DBC19/CONTROLFILE/current.273.1035154345' ORA-15078: ASM diskgroup was forcibly dismounted ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_ckpt_16216.trc (incident=50625): ORA-221 [] [] [] [] [] [] [] [] [] [] [] [] Incident details in: /u01/app/oracle/diag/rdbms/dbc19/DBC19/incident/incdir_50625/DBC19_ckpt_16216_i50625.trc 2020-03-22T17:18:49.294372+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_mz00_12148.trc: ORA-15078: ASM diskgroup was forcibly dismounted 2020-03-22T17:18:49.388685+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_mz00_12148.trc: ORA-00202: control file: '+DATA/DBC19/CONTROLFILE/current.273.1035154345' ORA-15078: ASM diskgroup was forcibly dismounted 2020-03-22T17:18:49.476458+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_mz00_12148.trc: ORA-00202: control file: '+DATA/DBC19/CONTROLFILE/current.273.1035154345' ORA-15078: ASM diskgroup was forcibly dismounted 2020-03-22T17:18:49.549783+01:00 Errors in file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_mz00_12148.trc: ORA-00204: error in reading (block 1, # blocks 1) of control file ORA-00202: control file: '+DATA/DBC19/CONTROLFILE/current.273.1035154345' ORA-15078: ASM diskgroup was forcibly dismounted USER (ospid: 16216): terminating the instance due to ORA error 221 2020-03-22T17:18:50.186640+01:00 System state dump requested by (instance=1, osid=16216 (CKPT)), summary=[abnormal instance termination]. error - 'Instance is terminating. ' System State dumped to trace file /u01/app/oracle/diag/rdbms/dbc19/DBC19/trace/DBC19_diag_16177.trc 2020-03-22T17:18:50.382616+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:50.388519+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:51.098190+01:00 Dumping diagnostic data in directory=[cdmp_20200322171850], requested by (instance=1, osid=16216 (CKPT)), summary=[abnormal instance termination]. 2020-03-22T17:18:51.362943+01:00 ORA-1092 : opitsk aborting process 2020-03-22T17:18:52.255137+01:00 Instance terminated by USER, pid = 16216 2020-03-22T17:18:52.275302+01:00 Warning: 2 processes are still attacheded to shmid 1179660: (size: 36864 bytes, creator pid: 16127, last attach/detach pid: 12191) 2020-03-22T17:18:52.404765+01:00 License high water mark = 10 2020-03-22T17:18:53.405607+01:00 USER(prelim) (ospid: 12191): terminating the instance 2020-03-22T17:18:53.407326+01:00 Instance terminated by USER(prelim), pid = 12191