RMAN, Allocate channel, CDB, and CLOSE: bug

Allocate channel for RMAN is used in various scenarios, most of the time is useful when you use tape as device type or you need to use some kind of format. The way to do the allocation not changed since a long time ago, but when you run the database in container mode you can hit a bug that turns your channel unusable. I will show you the bug and how to avoid it with a simple trick.

This bug hit every version since 12 and I discovered it last year when testing some scenarios, but I was able to test and post just recently. It just occurs for CDB databases and exists just one one-off solution published for 12.2. But there is one workaround more useful and works for every version.

The most interesting part is that everything that we made until now when allocate channel will not work. You can search in all doc available for allocate channel since 9i until 19c the first thing that you made after open the run{} is allocate channel. This is the default and recommended in the docs: for 19c, 18c, and 9i.

My environment for this post is 18c database (last update from April 2019) with just one PDB:

[oracle@orcloel7 ~]$ sqlplus / as sysdba

SQL*Plus: Release 18.0.0.0.0 - Production on Sun Jun 30 23:24:23 2019
Version 18.6.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.


Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.6.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 ORCL18P                        READ WRITE NO
SQL> exit
Disconnected from Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.6.0.0.0
[oracle@orcloel7 ~]$

And for rman I show the schema and the backup for one datafile (#12) just to have smaller output.

RMAN> report schema;

Report of database schema for database with db_unique_name ORCL18C

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    870      SYSTEM               YES     /u01/app/oracle/oradata/ORCL18C/system01.dbf
3    490      SYSAUX               NO      /u01/app/oracle/oradata/ORCL18C/sysaux01.dbf
4    330      UNDOTBS1             YES     /u01/app/oracle/oradata/ORCL18C/undotbs01.dbf
5    290      PDB$SEED:SYSTEM      NO      /u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf
6    340      PDB$SEED:SYSAUX      NO      /u01/app/oracle/oradata/ORCL18C/pdbseed/sysaux01.dbf
7    5        USERS                NO      /u01/app/oracle/oradata/ORCL18C/users01.dbf
8    100      PDB$SEED:UNDOTBS1    NO      /u01/app/oracle/oradata/ORCL18C/pdbseed/undotbs01.dbf
9    300      ORCL18P:SYSTEM       YES     /u01/app/oracle/oradata/ORCL18C/ORCL18P/system01.dbf
10   360      ORCL18P:SYSAUX       NO      /u01/app/oracle/oradata/ORCL18C/ORCL18P/sysaux01.dbf
11   100      ORCL18P:UNDOTBS1     YES     /u01/app/oracle/oradata/ORCL18C/ORCL18P/undotbs01.dbf
12   5        ORCL18P:USERS        NO      /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    37       TEMP                 32767       /u01/app/oracle/oradata/ORCL18C/temp01.dbf
2    62       PDB$SEED:TEMP        32767       /u01/app/oracle/oradata/ORCL18C/pdbseed/temp012019-05-25_23-51-33-086-PM.dbf
3    62       ORCL18P:TEMP         32767       /u01/app/oracle/oradata/ORCL18C/ORCL18P/temp01.dbf

RMAN> list backup of datafile 12 completed after "sysdate - 60/1440";

starting full resync of recovery catalog
full resync complete

List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ----------------
4610    Incr 1  40.00K     SBT_TAPE    00:00:02     02-07-2019_00-03
        BP Key: 4611   Status: AVAILABLE  Compressed: YES  Tag: TAG20190701T235918
        Handle: VB$_4142545763_4609I   Media:
  List of Datafiles in backup set 4610
  File LV Type Ckp SCN    Ckp Time         Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ---------------- ----------- ------ ----
  12   1  Incr 1714757    02-07-2019_00-03              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ----------------
4614    Incr 0  36.50K     SBT_TAPE    00:00:02     02-07-2019_00-03
        BP Key: 4615   Status: AVAILABLE  Compressed: YES  Tag: TAG20190701T235918
        Handle: VB$_4142545763_4609_12   Media:
  List of Datafiles in backup set 4614
  File LV Type Ckp SCN    Ckp Time         Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ---------------- ----------- ------ ----
  12   0  Incr 1714757    02-07-2019_00-03              NO    /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf

RMAN>

But look the example below where I try to restore the datafile (maybe because was corrupted – whatever the reason):

RMAN> run{
2> ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS  "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
RESTORE DATAFILE 12;
RECOVER DATAFILE 12;
}3> 4> 5> 6>

allocated channel: CH1
channel CH1: SID=72 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CA6E4CA5A751068E053010310AC0A07

Statement processed
starting full resync of recovery catalog
full resync complete

Starting restore at 02-07-2019_00-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=72 device type=DISK

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 07/02/2019 00:18:08
RMAN-06026: some targets not found - aborting restore
RMAN-06100: no channel to restore a backup or copy of datafile 12

RMAN>

And now this:

RMAN> run{
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS  "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
RESTORE DATAFILE 12;
RECOVER DATAFILE 12;
} 2> 3> 4> 5> 6>

Statement processed
starting full resync of recovery catalog
full resync complete

allocated channel: CH1
channel CH1: SID=63 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CA703A063E7127DE053010310ACC85B

Starting restore at 02-07-2019_00-18

channel CH1: starting datafile backup set restore
channel CH1: specifying datafile(s) to restore from backup set
channel CH1: restoring datafile 00012 to /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel CH1: reading from backup piece VB$_4142545763_4609_12
channel CH1: piece handle=VB$_4142545763_4609_12 tag=TAG20190701T235918
channel CH1: restored backup piece 1
channel CH1: restore complete, elapsed time: 00:00:25
Finished restore at 02-07-2019_00-19

Starting recover at 02-07-2019_00-19

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 02-07-2019_00-19
starting full resync of recovery catalog
full resync complete
released channel: CH1

RMAN> ALTER PLUGGABLE DATABASE ORCL18P OPEN INSTANCES=ALL;

Statement processed
starting full resync of recovery catalog
full resync complete

RMAN>

Saw the bug? YES, the bug occurs when you allocate the channel and after execute a close command. When you do the “close”, rman internally cleans everything related for the session (including allocated channels) and you can hit the “RMAN-06100: no channel to restore a backup” or others similar errors like RMAN-06012: channel: CH1 not allocated.

The bug is Bug 28076886 – RMAN: RMAN-06012: “channel: %s not allocated” after ALTER PLUGGABLE DATABASE CLOSE command (Doc ID 28076886.8).

The error occurs even for disk channels too (look full example):

RMAN> RUN {
ALLOCATE CHANNEL CH1 DEVICE TYPE DISK;
ALTER PLUGGABLE DATABASE ORCL18P CLOSE;
RECOVER DATABASE PREVIEW;
RELEASE CHANNEL CH1;
}2> 3> 4> 5> 6>

allocated channel: CH1
channel CH1: SID=110 device type=DISK

Statement processed
starting full resync of recovery catalog
full resync complete

Starting recover at 01-07-2019_00-34
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=110 device type=DISK
using channel ORA_DISK_1

no channel to restore a backup or copy of archived log for thread 1 with sequence 5 and starting SCN of 1552035
no channel to restore a backup or copy of archived log for thread 1 with sequence 6 and starting SCN of 1608615
no channel to restore a backup or copy of archived log for thread 1 with sequence 7 and starting SCN of 1669357
no channel to restore a backup or copy of archived log for thread 1 with sequence 8 and starting SCN of 1681533
no channel to restore a backup or copy of archived log for thread 1 with sequence 9 and starting SCN of 1681555
no channel to restore a backup or copy of archived log for thread 1 with sequence 10 and starting SCN of 1682610
List of Archived Log Copies for database with db_unique_name ORCL18C
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ----------------
3516    1    11      A 26-05-2019_19-13
        Name: /u01/app/oracle/oradata/ORCL18C/archivelog/2019_06_30/o1_mf_1_11_gklbz4oh_.arc

recovery will be done up to SCN 1603932
Media recovery start SCN is 1603932
Recovery must be done beyond SCN 1692620 to clear datafile fuzziness
Finished recover at 01-07-2019_00-35

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of release command at 07/01/2019 00:35:01
RMAN-06012: channel: CH1 not allocated

RMAN> RUN {
ALTER PLUGGABLE DATABASE ORCL18P CLOSE;
ALLOCATE CHANNEL CH1 DEVICE TYPE DISK;
RECOVER DATABASE PREVIEW;
RELEASE CHANNEL CH1;
}2> 3> 4> 5> 6>

Statement processed
starting full resync of recovery catalog
full resync complete

allocated channel: CH1
channel CH1: SID=110 device type=DISK

Starting recover at 01-07-2019_00-35

no channel to restore a backup or copy of archived log for thread 1 with sequence 5 and starting SCN of 1552035
no channel to restore a backup or copy of archived log for thread 1 with sequence 6 and starting SCN of 1608615
no channel to restore a backup or copy of archived log for thread 1 with sequence 7 and starting SCN of 1669357
no channel to restore a backup or copy of archived log for thread 1 with sequence 8 and starting SCN of 1681533
no channel to restore a backup or copy of archived log for thread 1 with sequence 9 and starting SCN of 1681555
no channel to restore a backup or copy of archived log for thread 1 with sequence 10 and starting SCN of 1682610
List of Archived Log Copies for database with db_unique_name ORCL18C
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ----------------
3516    1    11      A 26-05-2019_19-13
        Name: /u01/app/oracle/oradata/ORCL18C/archivelog/2019_06_30/o1_mf_1_11_gklbz4oh_.arc

recovery will be done up to SCN 1603932
Media recovery start SCN is 1603932
Recovery must be done beyond SCN 1693086 to clear datafile fuzziness
Finished recover at 01-07-2019_00-36

released channel: CH1

RMAN>

But can be trickier to discover the failure. Look the example below that I try to backup datafile #12:

RMAN> run{
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS  "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
BACKUP INCREMENTAL LEVEL 1 DATAFILE 12;
}2> 3> 4> 5>

allocated channel: CH1
channel CH1: SID=156 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CBA2C4600D309E8E053010310AC3E17

Statement processed
starting full resync of recovery catalog
full resync complete

Starting backup at 02-07-2019_23-25
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=106 device type=DISK
channel ORA_DISK_1: starting incremental level 1 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_DISK_1: starting piece 1 at 02-07-2019_23-25
channel ORA_DISK_1: finished piece 1 at 02-07-2019_23-25
piece handle=/u01/app/oracle/oradata/ORCL18C/89BE7E168E3D28BDE053010310AC2497/backupset/2019_07_02/o1_mf_nnnd1_TAG20190702T232547_gkqlyx8j_.bkp tag=TAG20190702T232547 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 02-07-2019_23-25

Starting Control File and SPFILE Autobackup at 02-07-2019_23-25
piece handle=/u01/app/oracle/oradata/ORCL18C/autobackup/2019_07_02/o1_mf_s_1012605951_gkqlz1vm_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 02-07-2019_23-25

RMAN> list backup of datafile 12 summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time  #Pieces #Copies Compressed Tag
------- -- -- - ----------- ---------------- ------- ------- ---------- ---
4270    B  0  A SBT_TAPE    01-07-2019_00-42 1       1       YES        TAG20190701T003820
4610    B  1  A SBT_TAPE    02-07-2019_00-03 1       1       YES        TAG20190701T235918
4614    B  0  A SBT_TAPE    02-07-2019_00-03 1       1       YES        TAG20190701T235918
5197    B  1  A DISK        02-07-2019_23-25 1       1       NO         TAG20190702T232547

RMAN>


And now:


RMAN> run{
ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
ALLOCATE CHANNEL CH1 DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS  "SBT_LIBRARY=/u01/app/oracle/product/18.6.0.0/dbhome_1/lib/libra.so, ENV=(RA_WALLET='location=file:/u01/app/oracle/product/18.6.0.0/dbhome_1/dbs/ra_zdlra credential_alias=zdlra-scan:1521/zdlra:orcl')";
BACKUP INCREMENTAL LEVEL 1 DATAFILE 12;
}2> 3> 4> 5>

Statement processed
starting full resync of recovery catalog
full resync complete

allocated channel: CH1
channel CH1: SID=105 device type=SBT_TAPE
channel CH1: RA Library (ZDLRA) SID=8CBAA68CCD751232E053010310AC374B

Starting backup at 02-07-2019_23-44
channel CH1: starting incremental level 1 datafile backup set
channel CH1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel CH1: starting piece 1 at 02-07-2019_23-44
channel CH1: finished piece 1 at 02-07-2019_23-44
piece handle=ORCL18C_3pu5ma33_1_1 tag=TAG20190702T234435 comment=API Version 2.0,MMS Version 12.2.0.2
channel CH1: backup set complete, elapsed time: 00:00:03
Finished backup at 02-07-2019_23-44

Starting Control File and SPFILE Autobackup at 02-07-2019_23-44
piece handle=c-551670416-20190702-02 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 02-07-2019_23-44
released channel: CH1

RMAN>

Look that because of the bug, in the first example, the backup was made to disk and not in the tape as requested. In the last, you can see that backup was made correctly in tape. But as you saw, no error was reported.

Conclusion

As you saw a tricky bug can change everything that we already know (and need to do) for a daily basis activity. One simple “close” for a PDB can destroy your backup/restore policy that you use from decades. Besides that even without “RMAN-06100: no channel to restore a backup” or other similar errors like RMAN-06012: channel: CH1 not allocated you can have problems.

Luckily if you are running affected version (everything below 19.2) the workaround is very simple to deploy.

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

Leave a Reply

Your email address will not be published. Required fields are marked *