Unable SSH to System After Installing Updates (CentOS/RHEL 7)

The Problem

Unable to connect to one of the systems after installing many software updates from the standard Oracle repositories. Unable to SSH/ping to the public IP and also unable to SSH/ping to the private IP via a different server in the same IP network.

The server console console looks healthy:

CentOS 7.8
Kernel 4.1.12-124.39.2.x86_64 on an x86_64

[Hostname] login:

The only change was performed the previous day - a yum update of all packages including the kernel followed by a system restart.

The Solution

Checking the /var/log/messages file for the period after the system update, it can be seen that the first error during the boot process is:

May 25 10:32:39 [HOSTNAME] dbus-daemon: /usr/bin/dbus-daemon: symbol lookup error: /usr/bin/dbus-daemon: undefined symbol: XML_SetHashSalt

This and subsequent errors are only present after the system update. The above error indicates that a new incompatibility exists between an installed and freshly updated library (libexpat.so.1) and a different version of the same library provided by a third party application.

The system library configuration needs to be edited to only use the operating system provided version of the library (libexpat.so.1). At a later stage, the third party application may be updated to also include a more recent version of the library, in which case the original configuration may be used again.

Steps to verify and correct the issue:

1. Remove the paths to any 3rd party libraries from /etc/ld.so.conf.d/. In the example below, a file named /etc/ld.so.conf.d/oracle.conf exists and contains a path to 3rd party libraries, including libexpat.so.1:

# cat /etc/ld.so.conf.d/oracle.conf
# ls -1 /soft/oracle/product/client/12.1/lib/libexpat.so*
# ls /usr/lib64/libexpat.so*

The incompatible library libexpat.so.1.5.2 takes precedence over the updated system library libexpat.so.1.6.0.

2. Move the application configuration file to a different location as a backup. Do not just rename the file.

# mv /etc/ld.so.conf.d/oracle.conf /[TEMPORARY_DIRECTORY]/

3. Rebuild the ldd cache:

# ldconfig

4. Verify that dbus now links against the expected library in /usr/lib64:

# ldd /usr/bin/dbus-daemon | grep libexpat
  libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f5000c0c000)

5. Reboot the system and test the boot process: