How to configure the system journal to preserve the record of events when a server is rebooted (CentOS/RHEL)
Storing the system journal permanently
By default, the system journals are kept in the /run/log/journal directory, which means the journals are cleared when the system reboots. You can change the conﬁguration settings of the systemd-journald service in the /etc/systemd/journald.conf ﬁle to make the journals persist across reboot.
The Storage parameter in the /etc/systemd/journald.conf ﬁle deﬁnes whether to store system journals in a volatile manner or persistently across reboot. Set this parameter to persistent, volatile, or auto as follows:
- persistent: stores journals in the /var/log/journal directory which persists across reboots. If the /var/log/journal directory does not exist, the systemd-journald service creates it.
- volatile: stores journals in the volatile /run/log/journal directory. As the /run ﬁle system is temporary and exists only in the runtime memory, data stored in it, including system journals, do not persist across reboot.
- auto: rsyslog determines whether to use persistent or volatile storage. If the /var/log/ journal directory exists, then rsyslog uses persistent storage, otherwise it uses volatile storage.
This is the default action if the Storage parameter is not set. The advantage of persistent system journals is that the historic data is available immediately at boot. However, even with a persistent journal, not all data is kept forever. The journal has a built-in log rotation mechanism that triggers monthly. In addition, by default, the journals are not allowed to get larger than 10% of the ﬁle system it is on, or leave less than 15% of the ﬁle system free. These values can be tuned for both the runtime and persistent journals in /etc/systemd/ journald.conf. The current limits on the size of the journal are logged when the systemdjournald process starts. The following command output shows the journal entries that reﬂect the current size limits:
[[email protected] ~]$ journalctl | grep -E 'Runtime|System journal' Feb 25 13:01:46 localhost systemd-journald: Runtime journal (/run/log/ journal/ae06db7da89142138408d77efea9229c) is 8.0M, max 91.4M, 83.4M free. Feb 25 13:01:48 remotehost.lab.example.com systemd-journald: Runtime journal (/run/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 91.4M, 83.4M free. Feb 25 13:01:48 remotehost.lab.example.com systemd-journald: System journal (/var/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 3.7G, 3.7G free. Feb 25 13:01:48 remotehost.lab.example.com systemd: Starting Tell Plymouth To Write Out Runtime Data... Feb 25 13:01:48 remotehost.lab.example.com systemd: Started Tell Plymouth To Write Out Runtime Data.
Conﬁguring Persistent System Journals
To conﬁgure the systemd-journald service to preserve system journals persistently across reboot, set Storage to persistent in the /etc/systemd/journald.conf ﬁle. Run the text editor of your choice as the superuser to edit the /etc/systemd/journald.conf ﬁle.
[Journal] Storage=persistent ...output omitted...
After editing the conﬁguration ﬁle, restart the systemd-journald service to bring the conﬁguration changes into effect.
[[email protected] ~]# systemctl restart systemd-journald
If the systemd-journald service successfully restarts, you can see that the /var/log/journal directory is created and contains one or more subdirectories. These subdirectories have hexadecimal characters in their long names and contain *.journal ﬁles. The *.journal ﬁles are the binary ﬁles that store the structured and indexed journal entries.
[[email protected] ~]# ls /var/log/journal 73ab164e278e48be9bf80e80714a8cd5 [[email protected] ~]# ls /var/log/journal/73ab164e278e48be9bf80e80714a8cd5 system.journal user-1000.journal
While the system journals persist across a reboot, you get an extensive number of entries in the output of the journalctl command that includes entries from the current system boot as well as the previous ones. To limit the output to a speciﬁc system boot, use the -b option with the journalctl command retrieves the entries limited to the ﬁrst system boot:
[[email protected] ~]# journalctl -b 1 ...output omitted...
The following journalctl command retrieves the entries limited to the second system boot. The following argument is meaningful only if the system has been rebooted for more than twice:
[[email protected] ~]# journalctl -b 2
The following journalctl command retrieves the entries limited to the current system boot:
[[email protected] ~]# journalctl -b