Recently I made one post about how to use the new feature -zeroDowntimeGIPatching when patching the Grid Infrastructure for 21c. It is a new feature/option that allows your database continues to be running while the grid is patched. You can see my post here. But during that post I talked about the usage of -updateosfiles when calling the rootcrs.sh and want to clarify some details and provide better examples.
Current environment
For this post, my environment is:
- OEL 8.4 Kernel 5.4.17-2102.201.3.el8uek.x86_64.
- Oracle GI 21c, version 21.5.
- Is a RAC installation, with two nodes.
The GI was upgraded from 21.3 to 21.5 as demonstrated in my post.
Compatibility Matrix
Before you think about upgrading the ACFS/AFD drivers you need to check if they are compatible with the version or kernel that you are running. The only place to check this is the MOS note ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1). On that note, you will see tables for each major version (18c, 19c, 21c), and you can see the versions of Linux Version and Kernel versions that are compatible. Below is marked for OEL 8:
And you can see that my version of Linux Kernel is compatible. If your version is not compatible, not update the ACFS/AFD kernel drivers.
ACFS and AFD Kernel drivers
From my previous post, you can see that I needed to use the skipDriverUpdate option when calling the root.sh to avoid the kernel drives being updated. This (having drivers updated) is not desired because your CRS will be shutdown and you will need to startup it manually, and of course, this will shutdown your databases too and will cause downtime (that is not the desire when you use the Zero-Downtime for GI patch).
But the point is, how do we discover if the GI patch will include or not new kernel drivers? The answer is not simple, or even, I don’t know and not found documentation with/of this information (like GI patch readme). So, my recommendation is simple: Always assume that the GI patch will deliver new ACFS and AFD drivers. So, always specify the skipDriverUpdate when calling the root.sh. In a simple way to avoid problems and unexpected downtimes.
So, before you need to discover the current kernel drivers versions. To do that you execute afddriverstate version or acfsdriverstate version:
[grid@oel8n1-21c ~]$ acfsdriverstate version ACFS-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. ACFS-9326: Driver build number = 210701. ACFS-9212: Driver build version = 21.0.0.0.0 (21.3.0.0.0). ACFS-9547: Driver available build number = 210701. ACFS-9548: Driver available build version = 21.0.0.0.0 (21.3.0.0.0). [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ /u01/app/21.0.0.0/grid/bin/afddriverstate version AFD-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. AFD-9326: Driver build number = 210701. AFD-9212: Driver build version = 21.0.0.0.0. AFD-9547: Driver available build number = 210701. AFD-9548: Driver available build version = 21.0.0.0.0. [grid@oel8n1-21c ~]$
Another option is to use the crsctl query driver activeversion or crsctl query driver softwareversion:
################################################################################## # #Check the current ACFS and AFD drivers version for all nodes # ################################################################################## [grid@oel8n1-21c ~]$ crsctl query driver activeversion -all Node Name : oel8n1-21c Driver Name : ACFS BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n1-21c Driver Name : AFD BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n2-21c Driver Name : ACFS BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n2-21c Driver Name : AFD BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ crsctl query driver softwareversion -all Node Name : oel8n1-21c Driver Name : ACFS BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n1-21c Driver Name : AFD BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n2-21c Driver Name : ACFS BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) Node Name : oel8n2-21c Driver Name : AFD BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) [grid@oel8n1-21c ~]$
With that, we have the current version which is 21.3 build version 21071.
ACFS and AFD Kernel drivers after GI Patch
So, after calling the root.sh with the skipDriverUpdate we will continue to have the old versions of the kernel drivers. Below, you can see that I am running GI 21.5 but the drivers still point to old ones:
[grid@oel8n1-21c ~]$ echo $ORACLE_HOME /u01/app/21.5.0.0/grid [grid@oel8n1-21c ~]$ [grid@oel8n1-21c ~]$ $ORACLE_HOME/OPatch/opatch lspatches 33693511;DBWLM RELEASE UPDATE 21.0.0.0.0 (33693511) 33556944;RHP RELEASE UPDATE 21.5.0.0.0 (33556944) 33531707;OCW RELEASE UPDATE 21.5.0.0.0 (33531707) 33531700;ACFS RELEASE UPDATE 21.5.0.0.0 (33531700) 33516412;Database Release Update : 21.5.0.0.220118 (33516412) 33276861;TOMCAT RELEASE UPDATE 21.0.0.0.0 (33276861) OPatch succeeded. [grid@oel8n1-21c ~]$ exit [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version ACFS-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. ACFS-9326: Driver build number = 211031. ACFS-9212: Driver build version = 21.0.0.0.0 (21.4.0.0.0). ACFS-9547: Driver available build number = 211031. ACFS-9548: Driver available build version = 21.0.0.0.0 (21.4.0.0.0). [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version AFD-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. AFD-9326: Driver build number = 210701. AFD-9212: Driver build version = 21.0.0.0.0. AFD-9547: Driver available build number = 211031. AFD-9548: Driver available build version = 21.0.0.0.0. [root@oel8n1-21c ~]#
Notice above (compared with the beginning of the post) that my ACFS was updated and my AFD no. This occurred because the patch process detected that I am not using ACFS (and just AFD), so, for this reason, it was updated.
And if I check the kernel modules files you will see that AFD was not touched (compare the date/time of oracleafd.ko with the others):
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko lrwxrwxrwx 1 root root 68 Mar 6 22:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko lrwxrwxrwx 1 root root 65 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko lrwxrwxrwx 1 root root 66 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko lrwxrwxrwx 1 root root 66 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko [root@oel8n1-21c ~]#
How to fix that?
So, the main question is how to fix that? And the only answer is using the rootcrs.sh -updateosfiles. But before that, some details.
If you saw my previous posts you will see that I inform you to do the reboot of the server after call the updateosfiles. This is wrong, you don’t need to do that.
The point is that calling the rootcrs.sh -updateosfiles will shutdown the CRS stack for the node and will not start it. You just need to start the CRS manually. Look below and you will see that I:
- Checked what was running.
- Called rootcrs.sh -updateosfiles.
- Check if the modules were loaded (it was).
- Check the time/date of the modules (was updated).
- Check CRS (nothing running).
- Start CRS.
- Later, everything was updated, and CRS running.
[root@oel8n1-21c ~]# date Sun Mar 27 17:52:21 CEST 2022 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep smon root 84975 1 1 17:28 ? 00:00:17 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 85618 1 0 17:29 ? 00:00:00 asm_smon_+ASM1 root 120081 5284 0 17:52 pts/0 00:00:00 grep --color=auto smon [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# date Sun Mar 27 17:52:29 CEST 2022 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/crs/install/rootcrs.sh -updateosfiles Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/grid/crsdata/oel8n1-21c/crsconfig/crsupdate_osfiles_oel8n1-21c_2022-03-27_05-52-47PM.log 2022/03/27 17:52:49 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service' [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# lsmod |grep oracle oracleacfs 5033984 0 oracleadvm 753664 0 oracleoks 765952 2 oracleadvm,oracleacfs oracleafd 221184 0 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko lrwxrwxrwx 1 root root 68 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko lrwxrwxrwx 1 root root 65 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep crs root 138660 5284 0 17:59 pts/0 00:00:00 grep --color=auto crs [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep crs root 141129 1 9 18:00 ? 00:00:03 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot grid 141370 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 141456 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 141474 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit grid 141499 1 1 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/crscdpd.bin root 142087 5284 0 18:01 pts/0 00:00:00 grep --color=auto crs [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver activeversion -all Node Name : oel8n1-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n1-21c Driver Name : AFD BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : AFD BuildNumber : 210701 BuildVersion : 21.0.0.0.0 (21.3.0.0.0) [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# date Sun Mar 27 18:02:34 CEST 2022 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep smon root 140078 1 1 18:00 ? 00:00:02 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 141620 1 0 18:00 ? 00:00:00 asm_smon_+ASM1 root 142893 5284 0 18:02 pts/0 00:00:00 grep --color=auto smon [root@oel8n1-21c ~]# ps -ef |grep lsnr grid 141370 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 141456 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 141474 1 0 18:00 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit root 142897 5284 0 18:02 pts/0 00:00:00 grep --color=auto lsnr [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# date Sun Mar 27 18:02:48 CEST 2022 [root@oel8n1-21c ~]#
So, we continue to have downtime because CRS went down, but is not needed to restart the entire node. Just start the CRS and everything will be ok. This is what I wanted to update from my previous post.
In the case of Kernel Update
So, if you need to upgrade/update the kernel of your Linux you (maybe) need to check and update the AFD and ACFS drivers. Rule #1 is always checking the MOS note ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1) and discover if your new kernel is compatible or not. If yes, you can update.
In the example below I made:
- Checked the currently running modules and versions.
- Stopped CRS (needed because I will reboot the Linux).
- Checked the available RPMs and the installed ones for Linux Kernel.
- Update the Linux Kernel to 5.4.17-2136.305.5.4.el8uek.
- Reboot server (if you use ksplice maybe is not needed, but I have not tested).
- Checked (after reboot that CRS is up and running).
- Check modules and versions for AFD and ACFS.
[root@oel8n1-21c ~]# uname -a Linux oel8n1-21c.oralocal 5.4.17-2102.201.3.el8uek.x86_64 #2 SMP Fri Apr 23 09:05:57 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# lsmod |grep oracle oracleacfs 5033984 0 oracleadvm 753664 0 oracleoks 765952 2 oracleadvm,oracleacfs oracleafd 221184 1 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko lrwxrwxrwx 1 root root 68 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko lrwxrwxrwx 1 root root 65 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version ACFS-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. ACFS-9326: Driver build number = 211031. ACFS-9212: Driver build version = 21.0.0.0.0 (21.4.0.0.0). ACFS-9547: Driver available build number = 211031. ACFS-9548: Driver available build version = 21.0.0.0.0 (21.4.0.0.0). [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version AFD-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. AFD-9326: Driver build number = 211031. AFD-9212: Driver build version = 21.0.0.0.0. AFD-9547: Driver available build number = 211031. AFD-9548: Driver available build version = 21.0.0.0.0. [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver activeversion -all Node Name : oel8n1-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n1-21c Driver Name : AFD BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : AFD BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver softwareversion -all Node Name : oel8n1-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n1-21c Driver Name : AFD BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : ACFS BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) Node Name : oel8n2-21c Driver Name : AFD BuildNumber : 211031 BuildVersion : 21.0.0.0.0 (21.4.0.0.0) [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep -v java |grep crs root 5098 1 4 18:49 ? 00:00:16 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot grid 6102 1 0 18:49 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 6434 1 0 18:50 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 6590 1 0 18:50 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit grid 6592 1 0 18:50 ? 00:00:01 /u01/app/21.5.0.0/grid/bin/crscdpd.bin grid 6616 1 0 18:50 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit grid 6618 1 0 18:50 ? 00:00:01 /u01/app/21.5.0.0/grid/bin/crscdpd.bin root 23147 7849 0 18:56 pts/0 00:00:00 grep --color=auto crs [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep smon root 3738 1 1 18:48 ? 00:00:05 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 4876 1 0 18:49 ? 00:00:00 asm_smon_+ASM1 oracle 8006 1 0 18:50 ? 00:00:00 ora_smon_orcl21c1 root 23151 7849 0 18:56 pts/0 00:00:00 grep --color=auto smon [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.crsd' on 'oel8n1-21c' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.orcl21c.db' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.cdp2.cdp' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.cdp1.cdp' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel8n1-21c' CRS-2677: Stop of 'ora.cdp2.cdp' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.cdp1.cdp' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.orcl21c.db' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel8n1-21c' CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.RECO.dg' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.cvu' on 'oel8n1-21c' CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.RECO.dg' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'oel8n1-21c' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.oel8n1-21c.vip' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel8n1-21c' CRS-2677: Stop of 'ora.asm' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel8n1-21c' CRS-2677: Stop of 'ora.oel8n1-21c.vip' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.scan1.vip' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.scan2.vip' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'oel8n1-21c' CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'oel8n1-21c' succeeded CRS-33677: Stop of resource group 'ora.asmgroup' on server 'oel8n1-21c' succeeded. CRS-2677: Stop of 'ora.cvu' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.qosmserver' on 'oel8n1-21c' succeeded CRS-2672: Attempting to start 'ora.qosmserver' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.cvu' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.oel8n1-21c.vip' on 'oel8n2-21c' CRS-2676: Start of 'ora.oel8n1-21c.vip' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.scan1.vip' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.scan2.vip' on 'oel8n2-21c' succeeded CRS-2672: Attempting to start 'ora.cdp1.cdp' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.cdp2.cdp' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel8n2-21c' CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel8n2-21c' CRS-2676: Start of 'ora.cvu' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.cdp1.cdp' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.cdp2.cdp' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel8n2-21c' succeeded CRS-2676: Start of 'ora.qosmserver' on 'oel8n2-21c' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'oel8n1-21c' CRS-2677: Stop of 'ora.ons' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'oel8n1-21c' CRS-2677: Stop of 'ora.net1.network' on 'oel8n1-21c' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel8n1-21c' has completed CRS-2677: Stop of 'ora.crsd' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.ctssd' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.evmd' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.asm' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel8n1-21c' CRS-2677: Stop of 'ora.drivers.acfs' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.ctssd' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.evmd' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.asm' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel8n1-21c' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'oel8n1-21c' CRS-2677: Stop of 'ora.cssd' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.crf' on 'oel8n1-21c' CRS-2673: Attempting to stop 'ora.driver.afd' on 'oel8n1-21c' CRS-2677: Stop of 'ora.driver.afd' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'oel8n1-21c' succeeded CRS-2677: Stop of 'ora.crf' on 'oel8n1-21c' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'oel8n1-21c' CRS-2677: Stop of 'ora.gipcd' on 'oel8n1-21c' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel8n1-21c' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# yum search kernel-uek ============================================================= Name Exactly Matched: kernel-uek ============================================================== kernel-uek.src : Oracle Unbreakable Enterprise Kernel Release 6 kernel-uek.x86_64 : Oracle Unbreakable Enterprise Kernel Release 6 ================================================================= Name Matched: kernel-uek ================================================================== kernel-uek-container.src : The Linux kernel optimized for running inside a container kernel-uek-container.x86_64 : The Linux kernel optimized for running inside a container kernel-uek-container-debug.x86_64 : Debug components for the kata-linux-container package. kernel-uek-debug.x86_64 : The Linux kernel compiled with extra debugging enabled kernel-uek-debug-devel.x86_64 : Development package for building kernel modules to match the debug kernel kernel-uek-devel.x86_64 : Development package for building kernel modules to match the kernel kernel-uek-doc.noarch : Various documentation bits found in the kernel source [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# rpm -qa |grep kernel kernel-headers-4.18.0-305.el8.x86_64 kernel-modules-4.18.0-305.el8.x86_64 kernel-tools-libs-4.18.0-305.el8.x86_64 kernel-4.18.0-305.el8.x86_64 kernel-core-4.18.0-305.el8.x86_64 kernel-uek-devel-5.4.17-2102.201.3.el8uek.x86_64 kernel-devel-4.18.0-305.el8.x86_64 kernel-uek-5.4.17-2102.201.3.el8uek.x86_64 kernel-tools-4.18.0-305.el8.x86_64 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# rpm -qa |grep linux-firmware linux-firmware-20201217-999.7.git7455a360.el8.noarch [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# yum update kernel-uek kernel-uek-devel linux-firmware Oracle Linux 8 BaseOS Latest (x86_64) 25 kB/s | 3.6 kB 00:00 Oracle Linux 8 BaseOS Latest (x86_64) 46 MB/s | 43 MB 00:00 Oracle Linux 8 Application Stream (x86_64) 75 kB/s | 3.9 kB 00:00 Oracle Linux 8 Application Stream (x86_64) 44 MB/s | 32 MB 00:00 Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 8 (x86_64) 42 kB/s | 3.0 kB 00:00 Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 8 (x86_64) 44 MB/s | 41 MB 00:00 Last metadata expiration check: 0:00:10 ago on Sun 03 Apr 2022 06:58:39 PM CEST. Dependencies resolved. ============================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================= Installing: kernel-uek x86_64 5.4.17-2136.305.5.4.el8uek ol8_UEKR6 109 M kernel-uek-devel x86_64 5.4.17-2136.305.5.4.el8uek ol8_UEKR6 18 M Upgrading: linux-firmware noarch 999:20211203-999.9.1.gitb0e898fb.el8 ol8_baseos_latest 206 M Transaction Summary ============================================================================================================================================================= Install 2 Packages Upgrade 1 Package Total download size: 333 M Is this ok [y/N]: y Downloading Packages: (1/3): kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64.rpm 12 MB/s | 18 MB 00:01 (2/3): kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64.rpm 23 MB/s | 109 MB 00:04 (3/3): linux-firmware-20211203-999.9.1.gitb0e898fb.el8.noarch.rpm 31 MB/s | 206 MB 00:06 ------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 50 MB/s | 333 MB 00:06 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Upgrading : linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch 1/4 Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 2/4 Installing : kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 2/4 Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 2/4 Installing : kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64 3/4 Running scriptlet: kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64 3/4 Cleanup : linux-firmware-999:20201217-999.7.git7455a360.el8.noarch 4/4 Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 4/4 ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored Running scriptlet: linux-firmware-999:20201217-999.7.git7455a360.el8.noarch 4/4 /sbin/ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored /sbin/ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored Verifying : kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 1/4 Verifying : kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64 2/4 Verifying : linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch 3/4 Verifying : linux-firmware-999:20201217-999.7.git7455a360.el8.noarch 4/4 Upgraded: linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch Installed: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64 kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64 Complete! [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# reboot login as: root root@10.160.10.70's password: Activate the web console with: systemctl enable --now cockpit.socket Last login: Sun Apr 3 18:54:39 2022 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# uname -a Linux oel8n1-21c.oralocal 5.4.17-2136.305.5.4.el8uek.x86_64 #2 SMP Mon Mar 28 19:05:22 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep -v java |grep crs root 6852 1 10 19:05 ? 00:00:07 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot grid 7223 1 0 19:05 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit grid 7267 1 0 19:05 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit grid 7275 1 0 19:05 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit grid 7290 1 0 19:05 ? 00:00:00 /u01/app/21.5.0.0/grid/bin/crscdpd.bin root 10199 1319 0 19:06 pts/0 00:00:00 grep --color=auto crs [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ps -ef |grep -v java |grep smon root 4207 1 2 19:04 ? 00:00:02 /u01/app/21.5.0.0/grid/bin/osysmond.bin grid 7622 1 0 19:05 ? 00:00:00 asm_smon_+ASM1 oracle 9478 1 0 19:06 ? 00:00:00 ora_smon_orcl21c1 root 10302 1319 0 19:06 pts/0 00:00:00 grep --color=auto smon [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# lsmod |grep oracle oracleacfs 5033984 0 oracleadvm 753664 0 oracleoks 765952 2 oracleadvm,oracleacfs oracleafd 221184 1 [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/oracle/oracleafd.ko lrwxrwxrwx 1 root root 68 Apr 3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleoks.ko lrwxrwxrwx 1 root root 65 Apr 3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleadvm.ko lrwxrwxrwx 1 root root 66 Apr 3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleacfs.ko lrwxrwxrwx 1 root root 66 Apr 3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version ACFS-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. ACFS-9326: Driver build number = 211031. ACFS-9212: Driver build version = 21.0.0.0.0 (21.4.0.0.0). ACFS-9547: Driver available build number = 211031. ACFS-9548: Driver available build version = 21.0.0.0.0 (21.4.0.0.0). [root@oel8n1-21c ~]# [root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version AFD-9325: Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64. AFD-9326: Driver build number = 211031. AFD-9212: Driver build version = 21.0.0.0.0. AFD-9547: Driver available build number = 211031. AFD-9548: Driver available build version = 21.0.0.0.0. [root@oel8n1-21c ~]#
But above you can see that after the Linux kernel update the same drivers AFD and ACFS continued installed. The GOOD part is that they got transferred correctly to the new kernel (because I updated all kernel rpm) and from GI perspective is ok. The BAD part is that I still need to call the rootcrs.sh -updateosfiles in case new versions are available.
In case (after the reboot) your CRS does not start, please check the logs, but probably you need to call the rootcrs.sh -updateosfiles to update the drivers. But remember to check if the kernel-uek-devel package was installed. It is important in case modules needed to be compiled for your kernel.
Conclusion
Again, if you reached here, Congrats. I know that was a long post with a lot of information. This post was made to show more details about the rootcrs.sh -updateosfiles and clarify some details. And also, to correct some information from my previous post when I wrote/told that was needed to reboot the server. It is not the case, you just need to startup CRS after calling the rootcrs.sh -updateosfiles.
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”
Pingback: 21c, Zero-Downtime Oracle Grid Infrastructure Patching - Fernando Simon
Pingback: 21c, Zero-Downtime Oracle Grid Infrastructure Patching – Silent Mode - Fernando Simon