Unable SSH to System After Installing Updates (CentOS/RHEL 7)
The Problem
Unable to connect to one of the systems after installing many software updates from the standard Oracle repositories. Unable to SSH/ping to the public IP and also unable to SSH/ping to the private IP via a different server in the same IP network.
The server console console looks healthy:
CentOS 7.8
Kernel 4.1.12-124.39.2.x86_64 on an x86_64
[Hostname] login:
The only change was performed the previous day - a yum update of all packages including the kernel followed by a system restart.
The Solution
Checking the /var/log/messages file for the period after the system update, it can be seen that the first error during the boot process is:
May 25 10:32:39 [HOSTNAME] dbus-daemon: /usr/bin/dbus-daemon: symbol lookup error: /usr/bin/dbus-daemon: undefined symbol: XML_SetHashSalt
This and subsequent errors are only present after the system update. The above error indicates that a new incompatibility exists between an installed and freshly updated library (libexpat.so.1) and a different version of the same library provided by a third party application.
The system library configuration needs to be edited to only use the operating system provided version of the library (libexpat.so.1). At a later stage, the third party application may be updated to also include a more recent version of the library, in which case the original configuration may be used again.
Steps to verify and correct the issue:
1. Remove the paths to any 3rd party libraries from /etc/ld.so.conf.d/. In the example below, a file named /etc/ld.so.conf.d/oracle.conf exists and contains a path to 3rd party libraries, including libexpat.so.1:
# cat /etc/ld.so.conf.d/oracle.conf
/soft/oracle/product/client/12.1/lib
# ls -1 /soft/oracle/product/client/12.1/lib/libexpat.so*
/soft/oracle/product/client/12.1/lib/libexpat.so
/soft/oracle/product/client/12.1/lib/libexpat.so.1
/soft/oracle/product/client/12.1/lib/libexpat.so.1.5.2
# ls /usr/lib64/libexpat.so*
/usr/lib64/libexpat.so.1
/usr/lib64/libexpat.so.1.6.0
The incompatible library libexpat.so.1.5.2 takes precedence over the updated system library libexpat.so.1.6.0.
2. Move the application configuration file to a different location as a backup. Do not just rename the file.
# mv /etc/ld.so.conf.d/oracle.conf /[TEMPORARY_DIRECTORY]/
3. Rebuild the ldd cache:
# ldconfig
4. Verify that dbus now links against the expected library in /usr/lib64:
# ldd /usr/bin/dbus-daemon | grep libexpat
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f5000c0c000)
5. Reboot the system and test the boot process: