{"id":859,"date":"2021-08-02T20:10:18","date_gmt":"2021-08-02T23:10:18","guid":{"rendered":"https:\/\/www.fernandosimon.com\/blog\/?p=859"},"modified":"2021-08-03T17:36:10","modified_gmt":"2021-08-03T20:36:10","slug":"fixing-exadata-missing-volumes-at-lvm","status":"publish","type":"post","link":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/","title":{"rendered":"Fixing Exadata Missing Volumes at LVM"},"content":{"rendered":"<p style=\"text-align: justify;\">Recently during the Exadata patch, one database node reported an issue during the <em>patchmgr<\/em> and stopped the patch apply. The error was related to missing volumes (LVDoNotRemoveOrUse) at LVM. During the post, you can check the error, but please take attention that it changes some LVM config file contents. So, check correctly the step executed and (if possible) open pro-active SR to be sure what you will be doing.<\/p>\n<p style=\"text-align: justify;\"><!--more Click here to read more...--><\/p>\n<p><a href=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-603 size-large\" src=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png\" alt=\"\" width=\"625\" height=\"588\" srcset=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png 1024w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-300x282.png 300w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-768x723.png 768w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-624x588.png 624w, https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png 1199w\" sizes=\"auto, (max-width: 625px) 100vw, 625px\" \/><\/a><\/p>\n<h1 style=\"text-align: justify;\">The error<\/h1>\n<p style=\"text-align: justify;\">During the patchmgr from nopde01 to node02 of dbnodes I got the error below:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm01s4 dbnodeupdate]# .\/dbserver_patch_20.210314\/patchmgr --dbnodes \/u01\/patches\/exadatapt\/dbnode_exavm_exavm02s4 --upgrade --iso_repo \/u01\/patches\/exadatapt\/domU\/p32459080_201000_Linux-x86-64.zip --target_version 20.1.8.0.0.210317 --skip_gi_db_validation\r\n\r\n************************************************************************************************************\r\nNOTE    patchmgr release: 21.210314 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)\r\nNOTE\r\nNOTE    Database nodes will reboot during the update process.\r\nNOTE\r\nWARNING Do not interrupt the patchmgr session.\r\nWARNING Do not resize the screen. It may disturb the screen layout.\r\nWARNING Do not reboot database nodes during update or rollback.\r\nWARNING Do not open logfiles in write mode and do not try to alter them.\r\n************************************************************************************************************\r\n2021-04-22 14:42:48 +0200        :INFO   : Checking hosts connectivity via ICMP\/ping\r\n2021-04-22 14:42:49 +0200        :INFO   : Hosts Reachable: [exavm02s4]\r\n2021-04-22 14:42:49 +0200        :INFO   : All hosts are reachable via ping\/ICMP\r\n2021-04-22 14:42:49 +0200        :Working: Verify SSH equivalence for the root user to exavm02s4\r\n2021-04-22 14:42:50 +0200        :INFO   : SSH equivalency verified to host exavm02s4\r\n2021-04-22 14:42:50 +0200        :SUCCESS: Verify SSH equivalence for the root user to exavm02s4\r\n2021-04-22 14:42:52 +0200        :Working: Initiate prepare steps on node(s).\r\n2021-04-22 14:42:53 +0200        :Working: Check free space on exavm02s4\r\n2021-04-22 14:42:57 +0200        :SUCCESS: Check free space on exavm02s4\r\n2021-04-22 14:43:23 +0200        :SUCCESS: Initiate prepare steps on node(s).\r\n2021-04-22 14:43:23 +0200        :Working: Initiate update on 1 node(s).\r\n2021-04-22 14:43:24 +0200        :Working: dbnodeupdate.sh running a backup on 1 node(s).\r\n2021-04-22 16:45:26 +0200        :ERROR  : dbnodeupdate.sh backup failed on one or more nodes\r\n\r\n    SUMMARY OF ERRORS FOR exavm02s4:\r\n\r\n    exavm02s4: ERROR: Backup failed investigate logfiles \/var\/log\/cellos\/dbnodeupdate.log and \/var\/log\/cellos\/dbserver_backup.sh.log\r\n\r\n\r\n2021-04-22 16:45:33 +0200        :FAILED : dbnodeupdate.sh running a backup on 1 node(s).\r\n[INFO     ] Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_upgrade_220421144247.tbz\r\n-rw-r--r-- 1 root root 1701698 Apr 22 16:45 Diag_patchmgr_dbnode_upgrade_220421144247.tbz\r\n2021-04-22 16:45:35 +0200        :ERROR  : FAILED run of command:.\/dbserver_patch_20.210314\/patchmgr --dbnodes \/u01\/patches\/exadatapt\/dbnode_exavm_exavm02s4 --upgrade --iso_repo \/u01\/patches\/exadatapt\/domU\/p32459080_201000_Linux-x86-64.zip --target_version 20.1.8.0.0.210317 --skip_gi_db_validation\r\n2021-04-22 16:45:35 +0200        :INFO   : Upgrade attempted on nodes in file \/u01\/patches\/exadatapt\/dbnode_exavm_exavm02s4: [exavm02s4]\r\n2021-04-22 16:45:35 +0200        :INFO   : Current image version on dbnode(s) is:\r\n2021-04-22 16:45:35 +0200        :INFO   : exavm02s4: 19.2.19.0.0.201013\r\n2021-04-22 16:45:35 +0200        :INFO   : For details, check the following files in \/u01\/patches\/exadatapt\/dbnodeupdate\/dbserver_patch_20.210314:\r\n2021-04-22 16:45:35 +0200        :INFO   :  - &lt;dbnode_name&gt;_dbnodeupdate.log\r\n2021-04-22 16:45:35 +0200        :INFO   :  - patchmgr.log\r\n2021-04-22 16:45:35 +0200        :INFO   :  - patchmgr.trc\r\n2021-04-22 16:45:35 +0200        :INFO   : Exit status:1\r\n2021-04-22 16:45:35 +0200        :INFO   : Exiting.\r\n<\/pre>\n<p style=\"text-align: justify;\">If you check the patchmgr.log we can check the same error message. But looking at <strong>\/var\/log\/cellos\/dbnodeupdate.log<\/strong> (at target node that will be patched) the true error appears:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# vi \/var\/log\/cellos\/dbnodeupdate.log\r\n...\r\n...\r\n    Setting interval between checks to 0 seconds\r\n    [INFO] Mount spare root partition \/dev\/VGExaDb\/LVDbSys2 to \/mnt_spare\r\n      Failed to find logical volume \"VGExaDb\/LVDoNotRemoveOrUse\"\r\n    [INFO] Preserve and then reset label for the root partition \/dev\/VGExaDb\/LVDbSys1\r\n    [INFO] Total amount of space available for snapshot: 1 GB\r\n    [INFO] Will be using snapshot of size: 1 GB\r\n    [INFO] Create LVM snapshot with 1 GB size of the root partition \/dev\/VGExaDb\/LVDbSys1\r\n      WARNING: Missing device \/dev\/xvda2 reappeared, updating metadata for VG VGExaDb to version 44.\r\n      WARNING: Device \/dev\/xvda2 still marked missing because of allocated data on it, remove volumes and consider vgreduce --removemissing.\r\n      WARNING: Missing device \/dev\/xvdd1 reappeared, updating metadata for VG VGExaDb to version 44.\r\n      WARNING: Device \/dev\/xvdd1 still marked missing because of allocated data on it, remove volumes and consider vgreduce --removemissing.\r\n      Cannot change VG VGExaDb while PVs are missing.\r\n      Consider vgreduce --removemissing.\r\n      Cannot process volume group VGExaDb\r\n    Unable to create LVM snapshot with 1Gb size of the root partition \/dev\/VGExaDb\/LVDbSys1\r\n[1619095405][2021-04-22 16:45:18 +0200][INFO][.\/dbnodeupdate.sh][DiaryEntry][]  Entering PrintGenError Backup failed investigate logfiles \/var\/log\/cellos\/dbnodeupdate.log and \/var\/log\/cellos\/dbserver_backup.sh.log\r\n...<\/pre>\n<p style=\"text-align: justify;\">The error is clear: \u201c<em>Missing device<\/em>\u201d and \u201c<em>Cannot change VG VGExaDb while PVs are missing<\/em>\u201d. So, basically, the LVM is reporting missing volumes and we need to recreate\/reimport it again.<\/p>\n<h1 style=\"text-align: justify;\">Recovering<\/h1>\n<p style=\"text-align: justify;\">The start point is to get a baseline from one correct node. If you don\u2019t know, one example from correct LVM volumes for Exadata VM are:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm01s4 dbserver_patch_20.210314]# pvs\r\n  PV         VG      Fmt  Attr PSize    PFree\r\n  \/dev\/xvda2 VGExaDb lvm2 a--   &lt;24.50g    0\r\n  \/dev\/xvdd1 VGExaDb lvm2 a--   &lt;62.00g    0\r\n  \/dev\/xvdf  VGExaDb lvm2 a--   &lt;50.00g    0\r\n  \/dev\/xvdg  VGExaDb lvm2 a--  &lt;150.00g    0\r\n[root@exavm01s4 dbserver_patch_20.210314]# vgs\r\n  VG      #PV #LV #SN Attr   VSize   VFree\r\n  VGExaDb   4   5   0 wz--n- 286.48g    0\r\n[root@exavm01s4 dbserver_patch_20.210314]#\r\n[root@exavm01s4 dbserver_patch_20.210314]# lvs\r\n  LV                 VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert\r\n  LVDbOra1           VGExaDb -wi-ao---- 221.48g\r\n  LVDbSwap1          VGExaDb -wi-ao----  16.00g\r\n  LVDbSys1           VGExaDb -wi-ao----  24.00g\r\n  LVDbSys2           VGExaDb -wi-a-----  24.00g\r\n  LVDoNotRemoveOrUse VGExaDb -wi-a-----   1.00g\r\n[root@exavm01s4 dbserver_patch_20.210314]#<\/pre>\n<p style=\"text-align: justify;\">But when I check the failed node I have:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# pvs\r\n  PV         VG      Fmt  Attr PSize    PFree\r\n  \/dev\/xvda2 VGExaDb lvm2 a-m   &lt;24.50g    0\r\n  \/dev\/xvdd1 VGExaDb lvm2 a-m   &lt;62.00g 1.00g\r\n  \/dev\/xvdf  VGExaDb lvm2 a--   &lt;50.00g    0\r\n  \/dev\/xvdg  VGExaDb lvm2 a--  &lt;150.00g    0\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# lvs\r\n  LV        VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert\r\n  LVDbOra1  VGExaDb -wi-ao--p- 221.48g\r\n  LVDbSwap1 VGExaDb -wi-ao--p-  16.00g\r\n  LVDbSys1  VGExaDb -wi-ao--p-  24.00g\r\n  LVDbSys2  VGExaDb -wi-a---p-  24.00g\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# vgs\r\n  VG      #PV #LV #SN Attr   VSize   VFree\r\n  VGExaDb   4   4   0 wz-pn- 286.48g 1.00g\r\n[root@exavm02s4 ~]#<\/pre>\n<p style=\"text-align: justify;\"><strong>As you can see the failed node is missing logical volumes, mainly the LVDoNotRemoveOrUse of 1GB.<\/strong><\/p>\n<p style=\"text-align: justify;\">The first step to solve is to try to remove the missing volumes. <strong>This step will fail but is crucial to be executed because it will generate one backup of the current configuration of LVM<\/strong>. And we need to edit this backup file to reload it.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# vgreduce --removemissing --verbose VGExaDb\r\n    There are 2 physical volumes missing.\r\n    There are 2 physical volumes missing.\r\n    Archiving volume group \"VGExaDb\" metadata (seqno 46).\r\n  WARNING: Partial LV LVDbSys1 needs to be repaired or removed.\r\n  WARNING: Partial LV LVDbSys2 needs to be repaired or removed.\r\n  WARNING: Partial LV LVDbOra1 needs to be repaired or removed.\r\n  WARNING: Partial LV LVDbSwap1 needs to be repaired or removed.\r\n  There are still partial LVs in VG VGExaDb.\r\n  To remove them unconditionally use: vgreduce --removemissing --force.\r\n  WARNING: Proceeding to remove empty missing PVs.\r\n    There are 2 physical volumes missing.\r\n    Creating volume group backup \"\/etc\/lvm\/backup\/VGExaDb\" (seqno 47).\r\n[root@exavm02s4 ~]#<\/pre>\n<p style=\"text-align: justify;\">Look that the file <strong>\/etc\/lvm\/backup\/VGExaDb<\/strong> is generated. And as an example, you can see that even trying to re-create or delete the missing volume generate error as well:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# lvcreate -n LVDoNotRemoveOrUse -L1G VGExaDb\r\n  WARNING: Missing device \/dev\/xvda2 reappeared, updating metadata for VG VGExaDb to version 48.\r\n  WARNING: Device \/dev\/xvda2 still marked missing because of allocated data on it, remove volumes and consider vgreduce --removemissing.\r\n  WARNING: Missing device \/dev\/xvdd1 reappeared, updating metadata for VG VGExaDb to version 48.\r\n  WARNING: Device \/dev\/xvdd1 still marked missing because of allocated data on it, remove volumes and consider vgreduce --removemissing.\r\n  Cannot change VG VGExaDb while PVs are missing.\r\n  Consider vgreduce --removemissing.\r\n  Cannot process volume group VGExaDb\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# vgextend --restoremissing LVDbSys1 VGExaDb\r\n  Volume group \"LVDbSys1\" not found\r\n  Cannot process volume group LVDbSys1\r\n[root@exavm02s4 ~]#<\/pre>\n<h1 style=\"text-align: justify;\">Modifying the LVM<\/h1>\n<p style=\"text-align: justify;\">After the \/etc\/lvm\/backup\/VGExaDb is created we can check the content of it and verify that are volumes (physical) marked with the MISSING flag:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# cd \/etc\/lvm\/\r\n[root@exavm02s4 lvm]#\r\n[root@exavm02s4 lvm]# ls -l backup\/\r\ntotal 4\r\n-rw------- 1 root root 3653 Apr 22 17:19 VGExaDb\r\n[root@exavm02s4 lvm]#\r\n[root@exavm02s4 lvm]#\r\n[root@exavm02s4 lvm]# cat backup\/VGExaDb\r\n# Generated by LVM2 version 2.02.186(2)-RHEL7 (2019-08-27): Thu Apr 22 17:19:30 2021\r\n\r\ncontents = \"Text Format Volume Group\"\r\nversion = 1\r\n\r\ndescription = \"Created *after* executing 'vgreduce --removemissing --verbose VGExaDb'\"\r\n\r\ncreation_host = \"exavm02s4.mynt.simon.net\"      # Linux exavm02s4.mynt.simon.net 4.1.12-124.42.4.el7uek.x86_64 #2 SMP Thu Sep 3 16:14:48 PDT 2020 x86_64\r\ncreation_time = 1619104770      # Thu Apr 22 17:19:30 2021\r\n\r\nVGExaDb {\r\n        id = \"ynfwGi-HZPF-0fe9-38lq-MbKE-DMhp-40QUXh\"\r\n        seqno = 48\r\n        format = \"lvm2\"                 # informational\r\n        status = [\"RESIZEABLE\", \"READ\", \"WRITE\"]\r\n        flags = []\r\n        extent_size = 8192              # 4 Megabytes\r\n        max_lv = 0\r\n        max_pv = 0\r\n        metadata_copies = 0\r\n\r\n        physical_volumes {\r\n\r\n                pv0 {\r\n                        id = \"o6OAXC-J3xd-Z8YT-mE2j-MZyN-UM4H-7cP3Q0\"\r\n                        device = \"\/dev\/xvda2\"   # Hint only\r\n\r\n                        status = [\"ALLOCATABLE\"]\r\n                        flags = [\"MISSING\"]\r\n                        dev_size = 51380126     # 24.5 Gigabytes\r\n                        pe_start = 384\r\n                        pe_count = 6271 # 24.4961 Gigabytes\r\n                }\r\n\r\n                pv1 {\r\n                        id = \"eQDYEs-cwbA-OI58-R9hP-Sure-bsbI-rE5w0x\"\r\n                        device = \"\/dev\/xvdd1\"   # Hint only\r\n\r\n                        status = [\"ALLOCATABLE\"]\r\n                        flags = [\"MISSING\"]\r\n                        dev_size = 130023326    # 62 Gigabytes\r\n                        pe_start = 384\r\n                        pe_count = 15871        # 61.9961 Gigabytes\r\n                }\r\n\r\n                pv2 {\r\n                        id = \"fE09JF-ajaK-7057-oiZL-6DRO-UGfL-l1vIFy\"\r\n                        device = \"\/dev\/xvdf\"    # Hint only\r\n\r\n                        status = [\"ALLOCATABLE\"]\r\n                        flags = []\r\n                        dev_size = 104857600    # 50 Gigabytes\r\n                        pe_start = 2048\r\n                        pe_count = 12799        # 49.9961 Gigabytes\r\n                }\r\n\r\n                pv3 {\r\n                        id = \"SNJiFM-PEjR-xuyU-8kBK-vqGf-b5AT-kDfW2S\"\r\n                        device = \"\/dev\/xvdg\"    # Hint only\r\n\r\n                        status = [\"ALLOCATABLE\"]\r\n                        flags = []\r\n                        dev_size = 314572800    # 150 Gigabytes\r\n                        pe_start = 2048\r\n                        pe_count = 38399        # 149.996 Gigabytes\r\n                }\r\n        }\r\n\r\n        logical_volumes {\r\n...\r\n...<\/pre>\n<p style=\"text-align: justify;\">So, we edit the file and remove the MISSING flag content (not the entire flag &#8211; just the value of it):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 lvm]# cat \/etc\/lvm\/backup\/VGExaDb |grep MISSING\r\n                        flags = [\"MISSING\"]\r\n                        flags = [\"MISSING\"]\r\n[root@exavm02s4 lvm]#\r\n[root@exavm02s4 lvm]#\r\n[root@exavm02s4 lvm]# cd \/etc\/lvm\/backup\/\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]# vi \/etc\/lvm\/backup\/VGExaDb\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]# cat VGExaDb |grep MISSING\r\n[root@exavm02s4 backup]#<\/pre>\n<p style=\"text-align: justify;\">After remove we can restore the config file that we edit (the backup file). <strong>Please BE AWARE that this can damage your LVM if you have not edited the correct file<\/strong>. Never use files generated from another node, the volume&#8217;s ids can be different. Below one parameter is the backup file VGExaDb itself, and the other one is the name of the volume group (VG). Both have the same name, for Exadata the VG is VGExaDb.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 backup]# vgcfgrestore -f VGExaDb VGExaDb\r\n  Volume group VGExaDb has active volume: LVDbSys2.\r\n  Volume group VGExaDb has active volume: LVDbSwap1.\r\n  Volume group VGExaDb has active volume: LVDbSys1.\r\n  Volume group VGExaDb has active volume: LVDbOra1.\r\n  WARNING: Found 4 active volume(s) in volume group \"VGExaDb\".\r\n  Restoring VG with active LVs, may cause mismatch with its metadata.\r\nDo you really want to proceed with restore of volume group \"VGExaDb\", while 4 volume(s) are active? [y\/n]: y\r\n  Restored volume group VGExaDb\r\n  Scan of VG VGExaDb from \/dev\/xvda2 found metadata seqno 49 vs previous 48.\r\n  Scan of VG VGExaDb from \/dev\/xvdd1 found metadata seqno 49 vs previous 48.\r\n  Scan of VG VGExaDb from \/dev\/xvdf found metadata seqno 49 vs previous 48.\r\n  Scan of VG VGExaDb from \/dev\/xvdg found metadata seqno 49 vs previous 48.\r\n[root@exavm02s4 backup]#<\/pre>\n<p style=\"text-align: justify;\">As you can see above the volume group was restored and a new sequence number was generated to identify it.<\/p>\n<p style=\"text-align: justify;\">After that we can scan the volumes again to check if everything was added correctly and reboot the node:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 backup]# vgscan\r\n  Reading volume groups from cache.\r\n  Found volume group \"VGExaDb\" using metadata type lvm2\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]# vgs\r\n  VG      #PV #LV #SN Attr   VSize   VFree\r\n  VGExaDb   4   4   0 wz--n- 286.48g 1.00g\r\n[root@exavm02s4 backup]#\r\n[root@exavm02s4 backup]# reboot\r\n...\r\n...<\/pre>\n<p style=\"text-align: justify;\">After the reboot we can scan again and <strong>recreate the missing volume LVDoNotRemoveOrUse<\/strong>:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">[root@exavm02s4 ~]# vgscan\r\n  Reading volume groups from cache.\r\n  Found volume group \"VGExaDb\" using metadata type lvm2\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# lvscan\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSys1' [24.00 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSys2' [24.00 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbOra1' [221.48 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSwap1' [16.00 GiB] inherit\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# lvcreate -n LVDoNotRemoveOrUse -L1G VGExaDb\r\n  Logical volume \"LVDoNotRemoveOrUse\" created.\r\n[root@exavm02s4 ~]#\r\n[root@exavm02s4 ~]# lvscan\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSys1' [24.00 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSys2' [24.00 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbOra1' [221.48 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDbSwap1' [16.00 GiB] inherit\r\n  ACTIVE            '\/dev\/VGExaDb\/LVDoNotRemoveOrUse' [1.00 GiB] inherit\r\n[root@exavm02s4 ~]#<\/pre>\n<p style=\"text-align: justify;\">Is not clear (and I couldn&#8217;t investigate) why this error appeared. it was not the first time that I got the same error. Maybe can be related to dracut issue that I described in <a href=\"https:\/\/www.fernandosimon.com\/blog\/exadata-dracut-and-lvm-locking_type-as-read-only\/\" target=\"_blank\" rel=\"noopener\">my previous post<\/a>.<\/p>\n<p style=\"text-align: justify;\"><strong>Disclaimer<\/strong>: <em>\u201cThe postings on this site are my own and don\u2019t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.\u201d<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently during the Exadata patch, one database node reported an issue during the patchmgr and stopped the patch apply. The error was related to missing volumes (LVDoNotRemoveOrUse) at LVM. During the post, you can check the error, but please take attention that it changes some LVM config file contents. So, check correctly the step executed [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[77,6,19,13],"tags":[100,69,162,160],"class_list":["post-859","post","type-post","status-publish","format-standard","hentry","category-engineeredsystems","category-exadata","category-linux","category-patchmgr","tag-engineered-systems","tag-exadata","tag-linux","tag-lvm"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Fixing Exadata Missing Volumes at LVM - Fernando Simon<\/title>\n<meta name=\"description\" content=\"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Fixing Exadata Missing Volumes at LVM - Fernando Simon\" \/>\n<meta property=\"og:description\" content=\"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\" \/>\n<meta property=\"og:site_name\" content=\"Fernando Simon\" \/>\n<meta property=\"article:published_time\" content=\"2021-08-02T23:10:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-03T20:36:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png\" \/>\n<meta name=\"author\" content=\"Simon\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Simon\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\"},\"author\":{\"name\":\"Simon\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"headline\":\"Fixing Exadata Missing Volumes at LVM\",\"datePublished\":\"2021-08-02T23:10:18+00:00\",\"dateModified\":\"2021-08-03T20:36:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\"},\"wordCount\":556,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png\",\"keywords\":[\"Engineered Systems\",\"Exadata\",\"linux\",\"LVm\"],\"articleSection\":[\"Engineered Systems\",\"Exadata\",\"Linux\",\"patchmgr\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\",\"name\":\"Fixing Exadata Missing Volumes at LVM - Fernando Simon\",\"isPartOf\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png\",\"datePublished\":\"2021-08-02T23:10:18+00:00\",\"dateModified\":\"2021-08-03T20:36:10+00:00\",\"author\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\"},\"description\":\"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png\",\"contentUrl\":\"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png\",\"width\":1199,\"height\":1129},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fernandosimon.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Fixing Exadata Missing Volumes at LVM\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#website\",\"url\":\"https:\/\/www.fernandosimon.com\/blog\/\",\"name\":\"Fernando Simon\",\"description\":\"Have you hugged your backup today?\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9\",\"name\":\"Simon\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g\",\"caption\":\"Simon\"},\"sameAs\":[\"http:\/\/www.fernandosimon.com\"],\"url\":\"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Fixing Exadata Missing Volumes at LVM - Fernando Simon","description":"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/","og_locale":"en_US","og_type":"article","og_title":"Fixing Exadata Missing Volumes at LVM - Fernando Simon","og_description":"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0","og_url":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/","og_site_name":"Fernando Simon","article_published_time":"2021-08-02T23:10:18+00:00","article_modified_time":"2021-08-03T20:36:10+00:00","og_image":[{"url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png","type":"","width":"","height":""}],"author":"Simon","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Simon","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#article","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/"},"author":{"name":"Simon","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"headline":"Fixing Exadata Missing Volumes at LVM","datePublished":"2021-08-02T23:10:18+00:00","dateModified":"2021-08-03T20:36:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/"},"wordCount":556,"commentCount":0,"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png","keywords":["Engineered Systems","Exadata","linux","LVm"],"articleSection":["Engineered Systems","Exadata","Linux","patchmgr"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/","url":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/","name":"Fixing Exadata Missing Volumes at LVM - Fernando Simon","isPartOf":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage"},"image":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2-1024x964.png","datePublished":"2021-08-02T23:10:18+00:00","dateModified":"2021-08-03T20:36:10+00:00","author":{"@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9"},"description":"Check all the steps that are needed to fix the missing LVM volumes that can cause errors during the Exadata patch.\u00a0","breadcrumb":{"@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#primaryimage","url":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png","contentUrl":"https:\/\/www.fernandosimon.com\/blog\/wp-content\/uploads\/2019\/09\/Exa-X8M-2.png","width":1199,"height":1129},{"@type":"BreadcrumbList","@id":"https:\/\/www.fernandosimon.com\/blog\/fixing-exadata-missing-volumes-at-lvm\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fernandosimon.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Fixing Exadata Missing Volumes at LVM"}]},{"@type":"WebSite","@id":"https:\/\/www.fernandosimon.com\/blog\/#website","url":"https:\/\/www.fernandosimon.com\/blog\/","name":"Fernando Simon","description":"Have you hugged your backup today?","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fernandosimon.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/386da956604bca0d5be5dd52210c1dd9","name":"Simon","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fernandosimon.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a3dbc48de62fffb1829befb4a588d789ec6dc5e05977afabb3407a5f37a16482?s=96&d=mm&r=g","caption":"Simon"},"sameAs":["http:\/\/www.fernandosimon.com"],"url":"https:\/\/www.fernandosimon.com\/blog\/author\/simon\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5ofTp-dR","_links":{"self":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/comments?post=859"}],"version-history":[{"count":0,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/posts\/859\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/media?parent=859"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/categories?post=859"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fernandosimon.com\/blog\/wp-json\/wp\/v2\/tags?post=859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}