2020-03-22T17:18:39.699555+01:00 WARNING: Write Failed. group:1 disk:1 AU:1 offset:4190208 size:4096 path:ORCL:CELLI01 incarnation:0xf0f0c1f3 asynchronous result:'I/O error' subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f9182f833d0 bufp:0x7f91836ef000 osderr1:0x3 osderr2:0x2e IO elapsed time: 0 usec Time waited on I/O: 0 usec WARNING: Hbeat write to PST disk 1.4042310131 in group 1 failed. [2] 2020-03-22T17:18:39.704035+01:00 NOTE: checking PST: grp = 1 2020-03-22T17:18:39.704361+01:00 GMON checking disk modes for group 1 at 132 for pid 46, osid 8102 2020-03-22T17:18:39.739691+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:39.740100+01:00 NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp 1 (DATA), dsk = 1/0xf0f0c1f3, mask = 0x6a, op = clear mandatory 2020-03-22T17:18:39.740483+01:00 GMON updating disk modes for group 1 at 133 for pid 46, osid 8102 WARNING: Write Failed. group:1 disk:1 AU:1 offset:4096 size:4096 path:ORCL:CELLI01 incarnation:0xf0f0c1f3 synchronous result:'I/O error' subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f9188353c78 bufp:0x7f9182f66000 osderr1:0x534b4950 osderr2:0x0 IO elapsed time: 0 usec Time waited on I/O: 0 usec NOTE: group DATA: updated PST location: disks 0008 0011 0000 2020-03-22T17:18:39.746945+01:00 NOTE: cache closing disk 9 of grp 1: (not open) CELLI03 ERROR: disk 1 (CELLI01) in group 1 (DATA) cannot be offlined because all disks [1(CELLI01), 9(CELLI03)] with mirrored data would be offline. 2020-03-22T17:18:39.747462+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:39.759171+01:00 NOTE: cache dismounting (not clean) group 1/0xB48031B9 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 12050, image: oracle@asmrec.oralocal (B001) 2020-03-22T17:18:39.761807+01:00 NOTE: halting all I/Os to diskgroup 1 (DATA) 2020-03-22T17:18:39.766289+01:00 NOTE: LGWR doing non-clean dismount of group 1 (DATA) thread 1 NOTE: LGWR sync ABA=23.3751 last written ABA 23.3751 2020-03-22T17:18:39.778273+01:00 NOTE: cache closing disk 9 of grp 1: (not open) CELLI03 2020-03-22T17:18:39.782954+01:00 WARNING: Offline of disk 1 (CELLI01) in group 1 and mode 0x7f failed on ASM inst 1 2020-03-22T17:18:39.791854+01:00 NOTE: initiating dirty detach from lock domain 1 2020-03-22T17:18:39.937916+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:40.089907+01:00 List of instances (total 1) : 1 My inst 1 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 2) Global Resource Directory partially frozen for dirty detach dirty detach - domain 1 invalid = TRUE 0 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete (total time 0.0 secs) 2020-03-22T17:18:40.093980+01:00 freeing rdom 1 freeing the fusion rht of pdb 1 2020-03-22T17:18:40.202205+01:00 WARNING: dirty detached from lock domain 1 NOTE: CKPT released recovery enqueue for thread 1 group 1 (DATA) 2020-03-22T17:18:40.206610+01:00 NOTE: cache dismounted group 1/0xB48031B9 (DATA) 2020-03-22T17:18:40.207406+01:00 SQL> alter diskgroup DATA dismount force /* ASM SERVER:3028300217 */ 2020-03-22T17:18:40.653641+01:00 NOTE: process _user8619_+asm1 (8619) initiating offline of disk 1.4042310131 (CELLI01) with mask 0x7e in group 1 (DATA) with client assisting NOTE: checking PST: grp = 1 2020-03-22T17:18:40.653887+01:00 GMON checking disk modes for group 1 at 134 for pid 51, osid 8619 2020-03-22T17:18:40.654007+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:40.654173+01:00 NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp 1 (DATA), dsk = 1/0xf0f0c1f3, mask = 0x6a, op = clear mandatory 2020-03-22T17:18:40.654803+01:00 GMON updating disk modes for group 1 at 135 for pid 51, osid 8619 2020-03-22T17:18:40.655039+01:00 NOTE: cache closing disk 9 of grp 1: (not open) CELLI03 ERROR: disk 1 (CELLI01) in group 1 (DATA) cannot be offlined because all disks [1(CELLI01), 9(CELLI03)] with mirrored data would be offline. 2020-03-22T17:18:40.655422+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:40.679181+01:00 WARNING: Offline of disk 1 (CELLI01) in group 1 and mode 0x7f failed on ASM inst 1 2020-03-22T17:18:40.835331+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:40.841818+01:00 ERROR: ORA-15130 in COD recovery for diskgroup 1/0xb48031b9 (DATA) ERROR: ORA-15130 thrown in RBAL for group number 1 2020-03-22T17:18:40.841979+01:00 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_8756.trc: ORA-15130: diskgroup "DATA" is being dismounted 2020-03-22T17:18:40.853738+01:00 NOTE: process _user12073_+asm1 (12073) initiating offline of disk 1.4042310131 (CELLI01) with mask 0x7e in group 1 (DATA) with client assisting NOTE: checking PST: grp = 1 2020-03-22T17:18:40.854161+01:00 GMON checking disk modes for group 1 at 136 for pid 53, osid 12073 2020-03-22T17:18:40.857287+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:40.857560+01:00 NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp 1 (DATA), dsk = 1/0xf0f0c1f3, mask = 0x6a, op = clear mandatory 2020-03-22T17:18:40.861120+01:00 GMON updating disk modes for group 1 at 137 for pid 53, osid 12073 2020-03-22T17:18:40.861317+01:00 NOTE: cache closing disk 9 of grp 1: (not open) CELLI03 ERROR: disk 1 (CELLI01) in group 1 (DATA) cannot be offlined because all disks [1(CELLI01), 9(CELLI03)] with mirrored data would be offline. 2020-03-22T17:18:40.861939+01:00 ERROR: too many offline disks in PST (grp 1) 2020-03-22T17:18:40.884878+01:00 WARNING: Offline of disk 1 (CELLI01) in group 1 and mode 0x7f failed on ASM inst 1 2020-03-22T17:18:41.128954+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:41.983353+01:00 NOTE: ASM client DBA19:DBA19:asmrec disconnected unexpectedly. NOTE: check client alert log. NOTE: cleaned up ASM client DBA19:DBA19:asmrec connection state (reg:1535110918) 2020-03-22T17:18:42.310634+01:00 Dumping diagnostic data in directory=[cdmp_20200322171842], requested by (instance=0, osid=980), summary=[trace bucket dump request (kfnclDelete0)]. 2020-03-22T17:18:43.081138+01:00 NOTE: detected orphaned ownerid 0x1000d; rcl client id 0x59987906b1d32b6b. NOTE: detected orphaned CGID 0x1000d. NOTE: detected orphaned CGID 0x1000c. 2020-03-22T17:18:43.214368+01:00 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_8756.trc: ORA-15130: diskgroup "DATA" is being dismounted 2020-03-22T17:18:43.214885+01:00 NOTE: client DBC19:DBC19:asmrec no longer has group 1 (DATA) mounted 2020-03-22T17:18:43.215492+01:00 NOTE: client DBB19:DBB19:asmrec no longer has group 1 (DATA) mounted NOTE: cache deleting context for group DATA 1/0xb48031b9 2020-03-22T17:18:43.256532+01:00 GMON dismounting group 1 at 138 for pid 48, osid 12050 2020-03-22T17:18:43.257029+01:00 NOTE: Disk CELLI04 in mode 0x7f marked for de-assignment NOTE: Disk CELLI01 in mode 0x7f marked for de-assignment NOTE: Disk CELLI07 in mode 0x7f marked for de-assignment NOTE: Disk CELLI06 in mode 0x7f marked for de-assignment NOTE: Disk CELLI02 in mode 0x7f marked for de-assignment NOTE: Disk CELLI03 in mode 0x1 marked for de-assignment NOTE: Disk CELLI05 in mode 0x7f marked for de-assignment 2020-03-22T17:18:43.259996+01:00 SUCCESS: diskgroup DATA was dismounted NOTE: Notify CRSD to check and update diskgroup resource state NOTE: diskgroup resource ora.DATA.dg state is update 2020-03-22T17:18:43.298551+01:00 SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:3028300217 */ SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DATA 2020-03-22T17:18:43.352003+01:00 SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:1:9} */ 2020-03-22T17:18:43.372816+01:00 NOTE: cache registered group DATA 1/0xB44031BF NOTE: cache began mount (first) of group DATA 1/0xB44031BF NOTE: Assigning number (1,8) to disk (ORCL:CELLI02) NOTE: Assigning number (1,0) to disk (ORCL:CELLI04) NOTE: Assigning number (1,11) to disk (ORCL:CELLI05) NOTE: Assigning number (1,3) to disk (ORCL:CELLI06) NOTE: Assigning number (1,2) to disk (ORCL:CELLI07) 2020-03-22T17:18:43.514642+01:00 cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1 2020-03-22T17:18:46.089517+01:00 NOTE: detected and added orphaned client id 0x10010 NOTE: detected and added orphaned client id 0x1000e 2020-03-22T17:18:47.412144+01:00 NOTE: opcode 32 request on group for client DBB19:DBB19:asmrec (reg:3981409293) gn=1 flags=0x0 2020-03-22T17:18:48.002643+01:00 NOTE: opcode 32 request on group for client DBC19:DBC19:asmrec (reg:1047050869) gn=1 flags=0x0 2020-03-22T17:18:49.097928+01:00 NOTE: detected and added orphaned client id 0x10010 NOTE: detected and added orphaned client id 0x1000e NOTE: released resources held for standalone CGID 0x1000e (gn: 1) NOTE: Proactively cleaning up dead client DBA19:DBA19:asmrec with orphan ownerid 0x59987906b1d32b6b; 6s elapsed WARNING: giving up on client id 0x59987906b1d32b6b [DBA19:DBA19:asmrec] which has not reconnected for 6 seconds (originally from ASM inst +ASM1, reg:1535110918) [dead] NOTE: removing stale ownerid 0x59987906b1d32b6b for client DBA19:DBA19:asmrec (reg:1535110918) NOTE: removing stale cgid 0x1000d for client DBA19:DBA19:asmrec (clientid:0x59987906b1d32b6b gn:2 startid:1035731938) NOTE: released resources held for CGID 0x1000d (gn: 2) NOTE: removing stale cgid 0x1000c for client DBA19:DBA19:asmrec (clientid:0x59987906b1d32b6b gn:1 startid:1035731938) NOTE: released resources held for CGID 0x1000c (gn: 1) NOTE: removing stale cgid 0x59987906b1d32b6b for client DBA19:DBA19:asmrec (clientid:0x59987906b1d32b6b gn:0 startid:1035731938) NOTE: released resources held for CGID 0x59987906b1d32b6b (gn: 0) NOTE: released resources held for client id 0x59987906b1d32b6b (reg:1535110918) NOTE: released resources held for standalone CGID 0x10010 (gn: 1) 2020-03-22T17:18:49.525018+01:00 NOTE: GMON heartbeating for grp 1 (DATA) GMON querying group 1 at 141 for pid 41, osid 10076 2020-03-22T17:18:49.526234+01:00 NOTE: ASM client DBB19:DBB19:asmrec disconnected unexpectedly. NOTE: check client alert log. NOTE: cleaned up ASM client DBB19:DBB19:asmrec connection state (reg:3981409293) 2020-03-22T17:18:49.559679+01:00 NOTE: Assigning number (1,1) to disk () NOTE: Assigning number (1,9) to disk () 2020-03-22T17:18:49.561152+01:00 GMON querying group 1 at 142 for pid 41, osid 10076 NOTE: initial disk modes for disk 9 () in group 1 (DATA) is not completely online: modes 0x1 lflags 0x0 2020-03-22T17:18:49.561350+01:00 NOTE: cache closing disk 9 of grp 1: (not open) 2020-03-22T17:18:49.568757+01:00 NOTE: cache dismounting (clean) group 1/0xB44031BF (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10076, image: oracle@asmrec.oralocal (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: LGWR not being messaged to dismount NOTE: cache dismounted group 1/0xB44031BF (DATA) NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xb44031bf NOTE: cache deleting context for group DATA 1/0xb44031bf 2020-03-22T17:18:49.570916+01:00 GMON dismounting group 1 at 143 for pid 41, osid 10076 2020-03-22T17:18:49.571050+01:00 NOTE: Disk CELLI04 in mode 0x7f marked for de-assignment NOTE: Disk CELLI07 in mode 0x7f marked for de-assignment NOTE: Disk CELLI06 in mode 0x7f marked for de-assignment NOTE: Disk CELLI02 in mode 0x7f marked for de-assignment NOTE: Disk CELLI05 in mode 0x7f marked for de-assignment ERROR: diskgroup DATA was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "9" is missing from group number "1" ORA-15042: ASM disk "1" is missing from group number "1" 2020-03-22T17:18:49.605727+01:00 ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:1:9} */ 2020-03-22T17:18:49.674485+01:00 SQL> ALTER DISKGROUP ALL MOUNT FOR dba19 /* asm agent *//* {0:10:40} *//* incarnation::2*/ NOTE: get diskgroups to mount for database 'DBA19' WARNING: failed to get diskgroup list for database 'DBA19' 2020-03-22T17:18:49.742580+01:00 SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:40} */ NOTE: cache registered group DATA 1/0xDC3031C2 NOTE: cache began mount (first) of group DATA 1/0xDC3031C2 2020-03-22T17:18:49.791288+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:49.833459+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:49.860082+01:00 Dumping diagnostic data in directory=[cdmp_20200322171849], requested by (instance=0, osid=13204), summary=[trace bucket dump request (kfnclDelete0)]. 2020-03-22T17:18:49.918195+01:00 NOTE: Assigning number (1,8) to disk (ORCL:CELLI02) NOTE: Assigning number (1,0) to disk (ORCL:CELLI04) NOTE: Assigning number (1,11) to disk (ORCL:CELLI05) NOTE: Assigning number (1,3) to disk (ORCL:CELLI06) NOTE: Assigning number (1,2) to disk (ORCL:CELLI07) 2020-03-22T17:18:49.925941+01:00 cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1 2020-03-22T17:18:51.990953+01:00 NOTE: ASM client DBC19:DBC19:asmrec disconnected unexpectedly. NOTE: check client alert log. NOTE: cleaned up ASM client DBC19:DBC19:asmrec connection state (reg:1047050869) 2020-03-22T17:18:52.106070+01:00 NOTE: detected and added orphaned client id 0x10011 NOTE: detected orphaned ownerid 0x1000f; rcl client id 0x8444026f69631300. NOTE: detected orphaned CGID 0x1000f. NOTE: client DBB19:DBB19:asmrec id 0xffffffffffffffff has reconnected to ASM inst +ASM1 (reg:3981409293), or has been fenced 2020-03-22T17:18:52.293131+01:00 Dumping diagnostic data in directory=[cdmp_20200322171852], requested by (instance=0, osid=16242), summary=[trace bucket dump request (kfnclDelete0)]. 2020-03-22T17:18:55.118691+01:00 NOTE: detected orphaned ownerid 0x10011; rcl client id 0x56e84d8e54299ba6. 2020-03-22T17:18:55.933809+01:00 NOTE: GMON heartbeating for grp 1 (DATA) GMON querying group 1 at 146 for pid 41, osid 10076 2020-03-22T17:18:55.935200+01:00 NOTE: Assigning number (1,1) to disk () NOTE: Assigning number (1,9) to disk () 2020-03-22T17:18:55.936113+01:00 GMON querying group 1 at 147 for pid 41, osid 10076 NOTE: initial disk modes for disk 9 () in group 1 (DATA) is not completely online: modes 0x1 lflags 0x0 2020-03-22T17:18:55.936393+01:00 NOTE: cache closing disk 9 of grp 1: (not open) 2020-03-22T17:18:55.936760+01:00 NOTE: cache dismounting (clean) group 1/0xDC3031C2 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10076, image: oracle@asmrec.oralocal (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: LGWR not being messaged to dismount NOTE: cache dismounted group 1/0xDC3031C2 (DATA) NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xdc3031c2 NOTE: cache deleting context for group DATA 1/0xdc3031c2 2020-03-22T17:18:55.937690+01:00 GMON dismounting group 1 at 148 for pid 41, osid 10076 2020-03-22T17:18:55.937879+01:00 NOTE: Disk CELLI04 in mode 0x7f marked for de-assignment NOTE: Disk CELLI07 in mode 0x7f marked for de-assignment NOTE: Disk CELLI06 in mode 0x7f marked for de-assignment NOTE: Disk CELLI02 in mode 0x7f marked for de-assignment NOTE: Disk CELLI05 in mode 0x7f marked for de-assignment ERROR: diskgroup DATA was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "9" is missing from group number "1" ORA-15042: ASM disk "1" is missing from group number "1" 2020-03-22T17:18:55.956011+01:00 ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:40} */ 2020-03-22T17:18:56.162103+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:56.218453+01:00 ASM Health Checker found 1 new failures 2020-03-22T17:18:58.122063+01:00 NOTE: Proactively cleaning up dead client DBB19:DBB19:asmrec with orphan ownerid 0x8444026f69631300; 6s elapsed WARNING: giving up on client id 0x8444026f69631300 [DBB19:DBB19:asmrec] which has not reconnected for 6 seconds (originally from ASM inst +ASM1, reg:3981409293) [dead] NOTE: removing stale ownerid 0x8444026f69631300 for client DBB19:DBB19:asmrec (reg:3981409293) NOTE: removing stale cgid 0x1000f for client DBB19:DBB19:asmrec (clientid:0x8444026f69631300 gn:2 startid:1035735787) NOTE: released resources held for CGID 0x1000f (gn: 2) NOTE: removing stale cgid 0x8444026f69631300 for client DBB19:DBB19:asmrec (clientid:0x8444026f69631300 gn:0 startid:1035735787) NOTE: released resources held for CGID 0x8444026f69631300 (gn: 0) NOTE: released resources held for client id 0x8444026f69631300 (reg:3981409293) 2020-03-22T17:18:59.002816+01:00 SQL> ALTER DISKGROUP ALL MOUNT FOR dbb19 /* asm agent *//* {0:10:42} *//* incarnation::3*/ 2020-03-22T17:18:59.002922+01:00 NOTE: get diskgroups to mount for database 'DBB19' WARNING: failed to get diskgroup list for database 'DBB19' 2020-03-22T17:18:59.078660+01:00 SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:42} */ NOTE: cache registered group DATA 1/0x169031C5 NOTE: cache began mount (first) of group DATA 1/0x169031C5 NOTE: Assigning number (1,8) to disk (ORCL:CELLI02) NOTE: Assigning number (1,0) to disk (ORCL:CELLI04) NOTE: Assigning number (1,11) to disk (ORCL:CELLI05) NOTE: Assigning number (1,3) to disk (ORCL:CELLI06) NOTE: Assigning number (1,2) to disk (ORCL:CELLI07) 2020-03-22T17:18:59.227368+01:00 cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1 2020-03-22T17:19:01.130894+01:00 NOTE: Proactively cleaning up dead client DBC19:DBC19:asmrec with orphan ownerid 0x56e84d8e54299ba6; 6s elapsed WARNING: giving up on client id 0x56e84d8e54299ba6 [DBC19:DBC19:asmrec] which has not reconnected for 6 seconds (originally from ASM inst +ASM1, reg:1047050869) [dead] NOTE: removing stale ownerid 0x56e84d8e54299ba6 for client DBC19:DBC19:asmrec (reg:1047050869) NOTE: removing stale cgid 0x10011 for client DBC19:DBC19:asmrec (clientid:0x56e84d8e54299ba6 gn:2 startid:1035736093) NOTE: released resources held for CGID 0x10011 (gn: 2) NOTE: removing stale cgid 0x56e84d8e54299ba6 for client DBC19:DBC19:asmrec (clientid:0x56e84d8e54299ba6 gn:0 startid:1035736093) NOTE: released resources held for CGID 0x56e84d8e54299ba6 (gn: 0) NOTE: released resources held for client id 0x56e84d8e54299ba6 (reg:1047050869) 2020-03-22T17:19:05.245199+01:00 NOTE: GMON heartbeating for grp 1 (DATA) GMON querying group 1 at 151 for pid 41, osid 10076 2020-03-22T17:19:05.246852+01:00 NOTE: Assigning number (1,1) to disk () NOTE: Assigning number (1,9) to disk () 2020-03-22T17:19:05.247244+01:00 GMON querying group 1 at 152 for pid 41, osid 10076 NOTE: initial disk modes for disk 9 () in group 1 (DATA) is not completely online: modes 0x1 lflags 0x0 2020-03-22T17:19:05.247625+01:00 NOTE: cache closing disk 9 of grp 1: (not open) 2020-03-22T17:19:05.248083+01:00 NOTE: cache dismounting (clean) group 1/0x169031C5 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10076, image: oracle@asmrec.oralocal (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: LGWR not being messaged to dismount NOTE: cache dismounted group 1/0x169031C5 (DATA) NOTE: cache ending mount (fail) of group DATA number=1 incarn=0x169031c5 NOTE: cache deleting context for group DATA 1/0x169031c5 2020-03-22T17:19:05.249084+01:00 GMON dismounting group 1 at 153 for pid 41, osid 10076 2020-03-22T17:19:05.249256+01:00 NOTE: Disk CELLI04 in mode 0x7f marked for de-assignment NOTE: Disk CELLI07 in mode 0x7f marked for de-assignment NOTE: Disk CELLI06 in mode 0x7f marked for de-assignment NOTE: Disk CELLI02 in mode 0x7f marked for de-assignment NOTE: Disk CELLI05 in mode 0x7f marked for de-assignment ERROR: diskgroup DATA was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "9" is missing from group number "1" ORA-15042: ASM disk "1" is missing from group number "1" 2020-03-22T17:19:05.268149+01:00 ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:42} */ 2020-03-22T17:19:05.327021+01:00 SQL> ALTER DISKGROUP ALL MOUNT FOR dbc19 /* asm agent *//* {0:10:44} *//* incarnation::2*/ NOTE: get diskgroups to mount for database 'DBC19' WARNING: failed to get diskgroup list for database 'DBC19' 2020-03-22T17:19:05.391290+01:00 SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:44} */ 2020-03-22T17:19:05.396160+01:00 ASM Health Checker found 1 new failures NOTE: cache registered group DATA 1/0x3E0031C8 NOTE: cache began mount (first) of group DATA 1/0x3E0031C8 NOTE: Assigning number (1,8) to disk (ORCL:CELLI02) NOTE: Assigning number (1,0) to disk (ORCL:CELLI04) NOTE: Assigning number (1,11) to disk (ORCL:CELLI05) NOTE: Assigning number (1,3) to disk (ORCL:CELLI06) NOTE: Assigning number (1,2) to disk (ORCL:CELLI07) 2020-03-22T17:19:05.537107+01:00 cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1 2020-03-22T17:19:11.548207+01:00 NOTE: GMON heartbeating for grp 1 (DATA) GMON querying group 1 at 156 for pid 41, osid 10076 2020-03-22T17:19:11.550741+01:00 NOTE: Assigning number (1,1) to disk () NOTE: Assigning number (1,9) to disk () 2020-03-22T17:19:11.551845+01:00 GMON querying group 1 at 157 for pid 41, osid 10076 NOTE: initial disk modes for disk 9 () in group 1 (DATA) is not completely online: modes 0x1 lflags 0x0 2020-03-22T17:19:11.552494+01:00 NOTE: cache closing disk 9 of grp 1: (not open) 2020-03-22T17:19:11.553979+01:00 NOTE: cache dismounting (clean) group 1/0x3E0031C8 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10076, image: oracle@asmrec.oralocal (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: LGWR not being messaged to dismount NOTE: cache dismounted group 1/0x3E0031C8 (DATA) NOTE: cache ending mount (fail) of group DATA number=1 incarn=0x3e0031c8 NOTE: cache deleting context for group DATA 1/0x3e0031c8 2020-03-22T17:19:11.557026+01:00 GMON dismounting group 1 at 158 for pid 41, osid 10076 2020-03-22T17:19:11.557406+01:00 NOTE: Disk CELLI04 in mode 0x7f marked for de-assignment NOTE: Disk CELLI07 in mode 0x7f marked for de-assignment NOTE: Disk CELLI06 in mode 0x7f marked for de-assignment NOTE: Disk CELLI02 in mode 0x7f marked for de-assignment NOTE: Disk CELLI05 in mode 0x7f marked for de-assignment ERROR: diskgroup DATA was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "9" is missing from group number "1" ORA-15042: ASM disk "1" is missing from group number "1" 2020-03-22T17:19:11.596536+01:00 ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {0:10:44} */