21c, updateosfiles after Grid Infrastructure Patch

Recently I made one post about how to use the new feature -zeroDowntimeGIPatching when patching the Grid Infrastructure for 21c. It is a new feature/option that allows your database continues to be running while the grid is patched. You can see my post here. But during that post I talked about the usage of -updateosfiles when calling the rootcrs.sh and want to clarify some details and provide better examples.

Current environment

For this post, my environment is:

  • OEL 8.4 Kernel 5.4.17-2102.201.3.el8uek.x86_64.
  • Oracle GI 21c, version 21.5.
  • Is a RAC installation, with two nodes.

The GI was upgraded from 21.3 to 21.5 as demonstrated in my post.

Compatibility Matrix

Before you think about upgrading the ACFS/AFD drivers you need to check if they are compatible with the version or kernel that you are running. The only place to check this is the MOS note ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1). On that note, you will see tables for each major version (18c, 19c, 21c), and you can see the versions of Linux Version and Kernel versions that are compatible. Below is marked for OEL 8:

And you can see that my version of Linux Kernel is compatible. If your version is not compatible, not update the ACFS/AFD kernel drivers.

ACFS and AFD Kernel drivers

From my previous post, you can see that I needed to use the skipDriverUpdate option when calling the root.sh to avoid the kernel drives being updated. This (having drivers updated) is not desired because your CRS will be shutdown and you will need to startup it manually, and of course, this will shutdown your databases too and will cause downtime (that is not the desire when you use the Zero-Downtime for GI patch).

But the point is, how do we discover if the GI patch will include or not new kernel drivers? The answer is not simple, or even, I don’t know and not found documentation with/of this information (like GI patch readme). So, my recommendation is simple: Always assume that the GI patch will deliver new ACFS and AFD drivers. So, always specify the skipDriverUpdate when calling the root.sh. In a simple way to avoid problems and unexpected downtimes.

So, before you need to discover the current kernel drivers versions. To do that you execute afddriverstate version or acfsdriverstate version:

[grid@oel8n1-21c ~]$ acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = 210701.
ACFS-9212:     Driver build version = 21.0.0.0.0 (21.3.0.0.0).
ACFS-9547:     Driver available build number = 210701.
ACFS-9548:     Driver available build version = 21.0.0.0.0 (21.3.0.0.0).
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$ /u01/app/21.0.0.0/grid/bin/afddriverstate version
AFD-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
AFD-9326:     Driver build number = 210701.
AFD-9212:     Driver build version = 21.0.0.0.0.
AFD-9547:     Driver available build number = 210701.
AFD-9548:     Driver available build version = 21.0.0.0.0.
[grid@oel8n1-21c ~]$

Another option is to use the crsctl query driver activeversion or crsctl query driver softwareversion:

##################################################################################
#
#Check the current ACFS and AFD drivers version for all nodes
#  
##################################################################################
[grid@oel8n1-21c ~]$ crsctl query driver activeversion -all
Node Name : oel8n1-21c
Driver Name : ACFS
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n1-21c
Driver Name : AFD
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n2-21c
Driver Name : ACFS
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n2-21c
Driver Name : AFD
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$ crsctl query driver softwareversion -all
Node Name : oel8n1-21c
Driver Name : ACFS
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n1-21c
Driver Name : AFD
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n2-21c
Driver Name : ACFS
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
Node Name : oel8n2-21c
Driver Name : AFD
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)
[grid@oel8n1-21c ~]$

With that, we have the current version which is 21.3 build version 21071.

ACFS and AFD Kernel drivers after GI Patch

So, after calling the root.sh with the skipDriverUpdate we will continue to have the old versions of the kernel drivers. Below, you can see that I am running GI 21.5 but the drivers still point to old ones:

[grid@oel8n1-21c ~]$ echo $ORACLE_HOME
/u01/app/21.5.0.0/grid
[grid@oel8n1-21c ~]$
[grid@oel8n1-21c ~]$ $ORACLE_HOME/OPatch/opatch lspatches
33693511;DBWLM RELEASE UPDATE 21.0.0.0.0 (33693511)
33556944;RHP RELEASE UPDATE 21.5.0.0.0 (33556944)
33531707;OCW RELEASE UPDATE 21.5.0.0.0 (33531707)
33531700;ACFS RELEASE UPDATE 21.5.0.0.0 (33531700)
33516412;Database Release Update : 21.5.0.0.220118 (33516412)
33276861;TOMCAT RELEASE UPDATE 21.0.0.0.0 (33276861)

OPatch succeeded.
[grid@oel8n1-21c ~]$ exit
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = 211031.
ACFS-9212:     Driver build version = 21.0.0.0.0 (21.4.0.0.0).
ACFS-9547:     Driver available build number = 211031.
ACFS-9548:     Driver available build version = 21.0.0.0.0 (21.4.0.0.0).
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version
AFD-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
AFD-9326:     Driver build number = 210701.
AFD-9212:     Driver build version = 21.0.0.0.0.
AFD-9547:     Driver available build number = 211031.
AFD-9548:     Driver available build version = 21.0.0.0.0.
[root@oel8n1-21c ~]#

Notice above (compared with the beginning of the post) that my ACFS was updated and my AFD no. This occurred because the patch process detected that I am not using ACFS (and just AFD), so, for this reason, it was updated.

And if I check the kernel modules files you will see that AFD was not touched (compare the date/time of oracleafd.ko with the others):

[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko
lrwxrwxrwx 1 root root 68 Mar  6 22:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko
lrwxrwxrwx 1 root root 65 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:28 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko
[root@oel8n1-21c ~]#

How to fix that?

So, the main question is how to fix that? And the only answer is using the rootcrs.sh -updateosfiles. But before that, some details.

If you saw my previous posts you will see that I inform you to do the reboot of the server after call the updateosfiles. This is wrong, you don’t need to do that.

The point is that calling the rootcrs.sh -updateosfiles will shutdown the CRS stack for the node and will not start it. You just need to start the CRS manually.  Look below and you will see that I:

  1. Checked what was running.
  2. Called rootcrs.sh -updateosfiles.
  3. Check if the modules were loaded (it was).
  4. Check the time/date of the modules (was updated).
  5. Check CRS (nothing running).
  6. Start CRS.
  7. Later, everything was updated, and CRS running.
[root@oel8n1-21c ~]# date
Sun Mar 27 17:52:21 CEST 2022
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep smon
root       84975       1  1 17:28 ?        00:00:17 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid       85618       1  0 17:29 ?        00:00:00 asm_smon_+ASM1
root      120081    5284  0 17:52 pts/0    00:00:00 grep --color=auto smon
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# date
Sun Mar 27 17:52:29 CEST 2022
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/crs/install/rootcrs.sh -updateosfiles
Using configuration parameter file: /u01/app/21.5.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel8n1-21c/crsconfig/crsupdate_osfiles_oel8n1-21c_2022-03-27_05-52-47PM.log
2022/03/27 17:52:49 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# lsmod |grep oracle
oracleacfs           5033984  0
oracleadvm            753664  0
oracleoks             765952  2 oracleadvm,oracleacfs
oracleafd             221184  0
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko
lrwxrwxrwx 1 root root 68 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko
lrwxrwxrwx 1 root root 65 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep crs
root      138660    5284  0 17:59 pts/0    00:00:00 grep --color=auto crs
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep crs
root      141129       1  9 18:00 ?        00:00:03 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot
grid      141370       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      141456       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      141474       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid      141499       1  1 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/crscdpd.bin
root      142087    5284  0 18:01 pts/0    00:00:00 grep --color=auto crs
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver activeversion -all
Node Name : oel8n1-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n1-21c
Driver Name : AFD
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : AFD
BuildNumber : 210701
BuildVersion : 21.0.0.0.0 (21.3.0.0.0)

[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# date
Sun Mar 27 18:02:34 CEST 2022
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep smon
root      140078       1  1 18:00 ?        00:00:02 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid      141620       1  0 18:00 ?        00:00:00 asm_smon_+ASM1
root      142893    5284  0 18:02 pts/0    00:00:00 grep --color=auto smon
[root@oel8n1-21c ~]# ps -ef |grep lsnr
grid      141370       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      141456       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      141474       1  0 18:00 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
root      142897    5284  0 18:02 pts/0    00:00:00 grep --color=auto lsnr
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# date
Sun Mar 27 18:02:48 CEST 2022
[root@oel8n1-21c ~]#

So, we continue to have downtime because CRS went down, but is not needed to restart the entire node. Just start the CRS and everything will be ok. This is what I wanted to update from my previous post.

In the case of Kernel Update

So, if you need to upgrade/update the kernel of your Linux you (maybe) need to check and update the AFD and ACFS drivers. Rule #1 is always checking the MOS note ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1) and discover if your new kernel is compatible or not. If yes, you can update.

In the example below I made:

  1. Checked the currently running modules and versions.
  2. Stopped CRS (needed because I will reboot the Linux).
  3. Checked the available RPMs and the installed ones for Linux Kernel.
  4. Update the Linux Kernel to 5.4.17-2136.305.5.4.el8uek.
  5. Reboot server (if you use ksplice maybe is not needed, but I have not tested).
  6. Checked (after reboot that CRS is up and running).
  7. Check modules and versions for AFD and ACFS.
[root@oel8n1-21c ~]# uname -a
Linux oel8n1-21c.oralocal 5.4.17-2102.201.3.el8uek.x86_64 #2 SMP Fri Apr 23 09:05:57 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# lsmod |grep oracle
oracleacfs           5033984  0
oracleadvm            753664  0
oracleoks             765952  2 oracleadvm,oracleacfs
oracleafd             221184  1
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko
lrwxrwxrwx 1 root root 68 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko
lrwxrwxrwx 1 root root 65 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko
lrwxrwxrwx 1 root root 66 Mar 27 17:54 /lib/modules/5.4.17-2102.201.3.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = 211031.
ACFS-9212:     Driver build version = 21.0.0.0.0 (21.4.0.0.0).
ACFS-9547:     Driver available build number = 211031.
ACFS-9548:     Driver available build version = 21.0.0.0.0 (21.4.0.0.0).
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version
AFD-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
AFD-9326:     Driver build number = 211031.
AFD-9212:     Driver build version = 21.0.0.0.0.
AFD-9547:     Driver available build number = 211031.
AFD-9548:     Driver available build version = 21.0.0.0.0.
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver activeversion -all
Node Name : oel8n1-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n1-21c
Driver Name : AFD
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : AFD
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl query driver softwareversion -all
Node Name : oel8n1-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n1-21c
Driver Name : AFD
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : ACFS
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

Node Name : oel8n2-21c
Driver Name : AFD
BuildNumber : 211031
BuildVersion : 21.0.0.0.0 (21.4.0.0.0)

[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep -v java |grep crs
root        5098       1  4 18:49 ?        00:00:16 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot
grid        6102       1  0 18:49 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid        6434       1  0 18:50 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid        6590       1  0 18:50 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
grid        6592       1  0 18:50 ?        00:00:01 /u01/app/21.5.0.0/grid/bin/crscdpd.bin
grid        6616       1  0 18:50 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid        6618       1  0 18:50 ?        00:00:01 /u01/app/21.5.0.0/grid/bin/crscdpd.bin
root       23147    7849  0 18:56 pts/0    00:00:00 grep --color=auto crs
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep smon
root        3738       1  1 18:48 ?        00:00:05 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid        4876       1  0 18:49 ?        00:00:00 asm_smon_+ASM1
oracle      8006       1  0 18:50 ?        00:00:00 ora_smon_orcl21c1
root       23151    7849  0 18:56 pts/0    00:00:00 grep --color=auto smon
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel8n1-21c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.orcl21c.db' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.cdp2.cdp' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.cdp1.cdp' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.cdp2.cdp' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.cdp1.cdp' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.orcl21c.db' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel8n1-21c'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.RECO.dg' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.RECO.dg' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.oel8n1-21c.vip' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.asm' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.oel8n1-21c.vip' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'oel8n1-21c' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'oel8n1-21c' succeeded.
CRS-2677: Stop of 'ora.cvu' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'oel8n1-21c' succeeded
CRS-2672: Attempting to start 'ora.qosmserver' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.cvu' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.oel8n1-21c.vip' on 'oel8n2-21c'
CRS-2676: Start of 'ora.oel8n1-21c.vip' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'oel8n2-21c' succeeded
CRS-2672: Attempting to start 'ora.cdp1.cdp' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.cdp2.cdp' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel8n2-21c'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel8n2-21c'
CRS-2676: Start of 'ora.cvu' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.cdp1.cdp' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.cdp2.cdp' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel8n2-21c' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'oel8n2-21c' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.ons' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.net1.network' on 'oel8n1-21c' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel8n1-21c' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.asm' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.cssd' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.crf' on 'oel8n1-21c'
CRS-2673: Attempting to stop 'ora.driver.afd' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.driver.afd' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel8n1-21c' succeeded
CRS-2677: Stop of 'ora.crf' on 'oel8n1-21c' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel8n1-21c'
CRS-2677: Stop of 'ora.gipcd' on 'oel8n1-21c' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel8n1-21c' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# yum search kernel-uek
============================================================= Name Exactly Matched: kernel-uek ==============================================================
kernel-uek.src : Oracle Unbreakable Enterprise Kernel Release 6
kernel-uek.x86_64 : Oracle Unbreakable Enterprise Kernel Release 6
================================================================= Name Matched: kernel-uek ==================================================================
kernel-uek-container.src : The Linux kernel optimized for running inside a container
kernel-uek-container.x86_64 : The Linux kernel optimized for running inside a container
kernel-uek-container-debug.x86_64 : Debug components for the kata-linux-container package.
kernel-uek-debug.x86_64 : The Linux kernel compiled with extra debugging enabled
kernel-uek-debug-devel.x86_64 : Development package for building kernel modules to match the debug kernel
kernel-uek-devel.x86_64 : Development package for building kernel modules to match the kernel
kernel-uek-doc.noarch : Various documentation bits found in the kernel source
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# rpm -qa |grep kernel
kernel-headers-4.18.0-305.el8.x86_64
kernel-modules-4.18.0-305.el8.x86_64
kernel-tools-libs-4.18.0-305.el8.x86_64
kernel-4.18.0-305.el8.x86_64
kernel-core-4.18.0-305.el8.x86_64
kernel-uek-devel-5.4.17-2102.201.3.el8uek.x86_64
kernel-devel-4.18.0-305.el8.x86_64
kernel-uek-5.4.17-2102.201.3.el8uek.x86_64
kernel-tools-4.18.0-305.el8.x86_64
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# rpm -qa |grep linux-firmware
linux-firmware-20201217-999.7.git7455a360.el8.noarch
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# yum update kernel-uek kernel-uek-devel linux-firmware
Oracle Linux 8 BaseOS Latest (x86_64)                                                                                         25 kB/s | 3.6 kB     00:00
Oracle Linux 8 BaseOS Latest (x86_64)                                                                                         46 MB/s |  43 MB     00:00
Oracle Linux 8 Application Stream (x86_64)                                                                                    75 kB/s | 3.9 kB     00:00
Oracle Linux 8 Application Stream (x86_64)                                                                                    44 MB/s |  32 MB     00:00
Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 8 (x86_64)                                                    42 kB/s | 3.0 kB     00:00
Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 8 (x86_64)                                                    44 MB/s |  41 MB     00:00
Last metadata expiration check: 0:00:10 ago on Sun 03 Apr 2022 06:58:39 PM CEST.
Dependencies resolved.
=============================================================================================================================================================
 Package                           Architecture            Version                                                  Repository                          Size
=============================================================================================================================================================
Installing:
 kernel-uek                        x86_64                  5.4.17-2136.305.5.4.el8uek                               ol8_UEKR6                          109 M
 kernel-uek-devel                  x86_64                  5.4.17-2136.305.5.4.el8uek                               ol8_UEKR6                           18 M
Upgrading:
 linux-firmware                    noarch                  999:20211203-999.9.1.gitb0e898fb.el8                     ol8_baseos_latest                  206 M

Transaction Summary
=============================================================================================================================================================
Install  2 Packages
Upgrade  1 Package

Total download size: 333 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64.rpm                                                                 12 MB/s |  18 MB     00:01
(2/3): kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64.rpm                                                                       23 MB/s | 109 MB     00:04
(3/3): linux-firmware-20211203-999.9.1.gitb0e898fb.el8.noarch.rpm                                                             31 MB/s | 206 MB     00:06
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         50 MB/s | 333 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                     1/1
  Upgrading        : linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch                                                                          1/4
  Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                        2/4
  Installing       : kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                        2/4
  Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                        2/4
  Installing       : kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                  3/4
  Running scriptlet: kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                  3/4
  Cleanup          : linux-firmware-999:20201217-999.7.git7455a360.el8.noarch                                                                            4/4
  Running scriptlet: kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                        4/4
ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored

  Running scriptlet: linux-firmware-999:20201217-999.7.git7455a360.el8.noarch                                                                            4/4
/sbin/ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored

/sbin/ldconfig: /etc/ld.so.conf.d/kernel-5.4.17-2102.201.3.el8uek.x86_64.conf:6: hwcap directive ignored

  Verifying        : kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                        1/4
  Verifying        : kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64                                                                                  2/4
  Verifying        : linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch                                                                          3/4
  Verifying        : linux-firmware-999:20201217-999.7.git7455a360.el8.noarch                                                                            4/4

Upgraded:
  linux-firmware-999:20211203-999.9.1.gitb0e898fb.el8.noarch
Installed:
  kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64                               kernel-uek-devel-5.4.17-2136.305.5.4.el8uek.x86_64

Complete!
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# reboot
login as: root
root@10.160.10.70's password:
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Sun Apr  3 18:54:39 2022
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# uname -a
Linux oel8n1-21c.oralocal 5.4.17-2136.305.5.4.el8uek.x86_64 #2 SMP Mon Mar 28 19:05:22 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep -v java |grep crs
root        6852       1 10 19:05 ?        00:00:07 /u01/app/21.5.0.0/grid/bin/crsd.bin reboot
grid        7223       1  0 19:05 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid        7267       1  0 19:05 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid        7275       1  0 19:05 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid        7290       1  0 19:05 ?        00:00:00 /u01/app/21.5.0.0/grid/bin/crscdpd.bin
root       10199    1319  0 19:06 pts/0    00:00:00 grep --color=auto crs
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ps -ef |grep -v java |grep smon
root        4207       1  2 19:04 ?        00:00:02 /u01/app/21.5.0.0/grid/bin/osysmond.bin
grid        7622       1  0 19:05 ?        00:00:00 asm_smon_+ASM1
oracle      9478       1  0 19:06 ?        00:00:00 ora_smon_orcl21c1
root       10302    1319  0 19:06 pts/0    00:00:00 grep --color=auto smon
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# lsmod |grep oracle
oracleacfs           5033984  0
oracleadvm            753664  0
oracleoks             765952  2 oracleadvm,oracleacfs
oracleafd             221184  1
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/oracle/oracleafd.ko
lrwxrwxrwx 1 root root 68 Apr  3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/oracle/oracleafd.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/oracle/oracleafd.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleoks.ko
lrwxrwxrwx 1 root root 65 Apr  3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleoks.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleoks.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleadvm.ko
lrwxrwxrwx 1 root root 66 Apr  3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleadvm.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleadvm.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# ls -l /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleacfs.ko
lrwxrwxrwx 1 root root 66 Apr  3 19:00 /lib/modules/5.4.17-2136.305.5.4.el8uek.x86_64/weak-updates/usm/oracleacfs.ko -> /lib/modules/5.4.17-2011.0.7.el8uek.x86_64/extra/usm/oracleacfs.ko
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# 
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = 211031.
ACFS-9212:     Driver build version = 21.0.0.0.0 (21.4.0.0.0).
ACFS-9547:     Driver available build number = 211031.
ACFS-9548:     Driver available build version = 21.0.0.0.0 (21.4.0.0.0).
[root@oel8n1-21c ~]#
[root@oel8n1-21c ~]# /u01/app/21.5.0.0/grid/bin/afddriverstate version
AFD-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
AFD-9326:     Driver build number = 211031.
AFD-9212:     Driver build version = 21.0.0.0.0.
AFD-9547:     Driver available build number = 211031.
AFD-9548:     Driver available build version = 21.0.0.0.0.
[root@oel8n1-21c ~]#

But above you can see that after the Linux kernel update the same drivers AFD and ACFS continued installed. The GOOD part is that they got transferred correctly to the new kernel (because I updated all kernel rpm) and from GI perspective is ok. The BAD part is that I still need to call the rootcrs.sh -updateosfiles in case new versions are available.

In case (after the reboot) your CRS does not start, please check the logs, but probably you need to call the rootcrs.sh -updateosfiles to update the drivers. But remember to check if the kernel-uek-devel package was installed. It is important in case modules needed to be compiled for your kernel.

Conclusion

Again, if you reached here, Congrats. I know that was a long post with a lot of information. This post was made to show more details about the rootcrs.sh -updateosfiles and clarify some details. And also, to correct some information from my previous post when I wrote/told that was needed to reboot the server. It is not the case, you just need to startup CRS after calling the rootcrs.sh -updateosfiles.

 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purposes, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

 

2 thoughts on “21c, updateosfiles after Grid Infrastructure Patch

  1. Pingback: 21c, Zero-Downtime Oracle Grid Infrastructure Patching - Fernando Simon

  2. Pingback: 21c, Zero-Downtime Oracle Grid Infrastructure Patching – Silent Mode - Fernando Simon

Leave a Reply

Your email address will not be published.