Recently in the CentOS Infra, we got a new sponsor giving us access to a server that has a HBA needing a kernel module that was deprecated in the RHEL8 (and thus CentOS 8) kernel by default.
What can you do in such situation ? Answer is easy : Elrepo ! They provide (for years now) kernel modules ready to go for network cards, raid/hba controllers, wifi nics, etc, and for various versions of RHEL/CentOS and other rebuilds using same kernel.
I wanted to give it a try on a node I have at least remote KVM/ipmi access, to reset the node in case of problem. Let's use the following ~8y old IBM blade for this example , that has the following network interface card and also hba :
# lspci |egrep -i 'ethernet|Serial'
0c:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
16:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 03)
# lspci -n |egrep '0c:00.0|16:00.0'
0c:00.0 0107: 1000:0070 (rev 03)
16:00.0 0200: 19a2:0710 (rev 03)
# ethtool -i eth0|grep driver
driver: be2net
# modinfo be2net|grep 0710
alias: pci:v000019A2d00000710sv*sd*bc*sc*i*
# lsmod|grep sas
mpt2sas 249763 2
# modinfo mpt2sas|grep 0070
alias: pci:v00001000d00000070sv*sd*bc*sc*i*
As you can see above, we were searching for the kernel module in use, and compare in kernel module the pci id. We know which kmod to search for and which pci id the kmod is supposed to support. Let's verify this in CentOS 8 : Let's try with network module :
# modinfo be2net|grep 0710 || echo "Sorry, doesn't seem supported"
Sorry, doesn't seem supported
Ouch, be2net is present but doesn't support our pci id so it was deprecated ... we need a different one. Let's try now with the hba :
# modinfo mpt2sas|grep 0070 || echo "Sorry, doesn't seem supported"
Sorry, doesn't seem supported
empty so also not supported. By chance, Elrepo has this packaged as rpm :
- http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64.rpm
- http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64.rpm
Of course we can also use a DuD as Elrepo already provides such .iso images.
In our case though, we have to do things differently, as we need two kmods and also no way to fetch it through network ( Obviously, as we need first nic kernel module/driver ) ...
So here was my idea :
- build a DuD .iso that has both kmods/kernel modules
- inject that .iso inside the initrd.img (as we need kernel module loaded before we can reach network for stage2 and no way to grab network driver through network obviously)
Let's go back on the CentOS 7 node that needs to be reinstalled with CentOS 8 :
# yum install -y genisoimage createrepo_c
# cd $(mktemp -d)
# mkdir -p {./dd/rpms/x86_64/,./dd/src}
# echo -e "Driver Update Disk version 3\c" > ./dd/rhdd3
# pushd dd/rpms/x86_64/
# wget http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/{kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64.rpm,kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64.rpm}
# createrepo_c ./
# popd
# pushd dd/src
# wget http://elrepo.reloumirrors.net/elrepo/el8/SRPMS/{kmod-be2net-12.0.0.0-5.el8_2.elrepo.src.rpm,kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.src.rpm}
# popd
# mkisofs -quiet -lR -V OEMDRV -input-charset utf8 -o mpt3sas-be2net-kmod.iso ./dd
Now that we have mpt3sas-be2net-kmod.iso we can use it with inst.dd= .. but as we have no network, anaconda needs to find it early in the process. So let's inject it into initrd.img ( you can do the same with kickstart )
Let's retrieve vmlinuz and initrd.img to remotely kick a CentOS 8 reinstall on itself (node is actually running CentOS 7)
# pushd /boot
# mirror_url="http://mirror.centos.org/centos/8/"
# curl --location --fail ${mirror_url}/BaseOS/x86_64/os/images/pxeboot/initrd.img > initrd.img.install
# curl --location --fail ${mirror_url}/BaseOS/x86_64/os/images/pxeboot/vmlinuz > vmlinuz.install
# popd
# echo mpt3sas-be2net-kmod.iso |cpio -c -o >> /boot/initrd.img.install
2005 blocks
Now that we have injected .iso into initrd.img, we can reference it for anaconda/install process as /<name>.iso.
Let's then just use kexec (usual) to remotely launch the install and use also VNC to see if everything is working : network should respond and be configured, and then we'll be able to see storage too.
# pushd /boot
# yum install -y kexec-tools
# kexec -l vmlinuz.install --append="net.ifnames=0 biosdevname=0 ksdevice=eth2 inst.repo=http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ inst.lang=en_GB inst.keymap=be-latin1 inst.dd=/mpt3sas-be2net-kmod.iso inst.vnc inst.vncpassword=DuDTest ip=172.22.0.16 netmask=255.255.254.0 gateway=172.22.1.254 nameserver=172.22.0.1 hostname=test.ci.centos.org pcie_aspm=off" --initrd=initrd.img.install && kexec -e
From that point it is like described in the previous link about kexec and kick a reinstall : kernel boots, loads initrd.img (but this time we see the DuD iso image being loaded and then it starts anaconda as usual. We can from there connect over vnc to finish the install (we have network and hba kernel module loaded and able to configure hardware)
Once machine is installed and rebooted, we can just ssh into it and clearly we can see that both rpm/kmods were installed ok (otherwise, no network nor storage and of course no install :) )
# modinfo mpt3sas|egrep 'filename|signer'
filename: /lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko
signer: ELRepo.org Secure Boot Key
# modinfo be2net|egrep 'filename|signer'
filename: /lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/be2net/be2net.ko
signer: ELRepo.org Secure Boot Key
# rpm -qa|grep elrepo
kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64
kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64
# yum installl -y elrepo-release
Of course, as shown above, don't forget to also install elrepo-release pkg, to then access newer kmods when needed, in case of a rebase between major.minor releases.
Hope you found that useful in case you need to upgrade working hardware but with deprecated drivers in the CentOS 8 kernel.