The operating systems break and crash. We as sysadmin are always looking for the root cause analysis so as to avoid such failures in the future. In this post, we will look at the basic troubleshooting of booting issues. This is a high-level overview and things can get pretty deep depending on the booting issue.
Looking at the logs of previously failed boots can be useful. If the system journals are persistent across reboots, you can use the journalctl tool to inspect those logs.
Remember that by default, the system journals are kept in the /run/log/journal directory, which means the journals are cleared when the system reboots. To store journals in the /var/log/journal directory, which persists across reboots, set the Storage parameter to persistent in /etc/systemd/journald.conf.
[root@host ~]# vim /etc/systemd/journald.conf .. [Journal] Storage=persistent ...
[root@host ~]# systemctl restart systemd-journald.service
To inspect the logs of a previous boot, use the -b option of journalctl. Without any arguments, the -b option only displays messages since the last boot. With a negative number as an argument, it displays the logs of previous boots.
[root@host ~]# journalctl -b -1 -p err
This command shows all messages rated as an error or worse from the previous boot.
Repairing systemd boot issue
To troubleshoot service startup issues at boot time, Red Hat Enterprise Linux 8 makes the following tools available.
Enabling the Early Debug Shell
By enabling the debug-shell service with systemctl enable debug-shell.service, the system spawns a root shell on TTY9 (Ctrl+Alt+F9) early during the boot sequence. This shell is automatically logged in as root, so that administrators can debug the system while the operating system is still booting.
Using the Emergency and Rescue Targets
By appending either systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line from the boot loader, the system spawns into a rescue or emergency shell instead of starting normally. Both of these shells require the root password.
The emergency target keeps the root file system mounted read-only, while the rescue target waits for sysinit.target to complete, so that more of the system is initialized, such as the logging service or the file systems. The root user at this point can not make changes to /etc/fstab until the drive is remounted in a read write state mount -o remount,rw /
Administrators can use these shells to fix any issues that prevent the system from booting normally; for example, a dependency loop between services, or an incorrect entry in /etc/fstab. Exiting from these shells continues with the regular boot process.
Identifying Stuck Jobs
During startup, systemd spawns a number of jobs. If some of these jobs cannot complete, they block other jobs from running. To inspect the current job list, administrators can use the systemctl list-jobs command. Any jobs listed as running must complete before the jobs listed as waiting can continue.