How to use journalctl to view log messages in journal for system troubleshooting

Finding Events

The systemd-journald service stores logging data in a structured, indexed binary ﬁle called the journal. This data includes extra information about the log event. For example, for syslog events, this includes the facility and the priority of the original message.

Note: In CentOS/RHEL 8, the /run/log directory stores the system journal by default. The contents of the /run/log directory get cleared after a reboot. You can change this setting if you want.

To retrieve log messages from the journal, use the journalctl command. You can use this command to view all messages in the journal or to search for speciﬁc events based on a wide range of options and criteria. If you run the command as root, you have full access to the journal. Regular users can also use this command but might be restricted from seeing certain messages.

[root@host ~]# journalctl
...output omitted...
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Stopped target Sockets.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Closed D-Bus User Message Bus Socket.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Closed Multimedia System.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Reached target Shutdown.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Starting Exit the Session...
Feb 21 17:46:25 host.lab.example.com systemd[24268]: pam_unix(systemduser:session): session c>
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopped User Manager for UID 1001.
Feb 21 17:46:25 host.lab.example.com systemd[1]: [email protected]: Unit not neede>
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopping /run/user/1001 mount wrapper...
Feb 21 17:46:25 host.lab.example.com systemd[1]: Removed slice User Slice of UID 1001.
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopped /run/user/1001 mount wrapper.
Feb 21 17:46:36 host.lab.example.com sshd[24434]: Accepted publickey for root from 172.25.250.>
Feb 21 17:46:37 host.lab.example.com systemd[1]: Started Session 20 of user root.
Feb 21 17:46:37 host.lab.example.com systemd-logind[708]: New session 20 of user root.
Feb 21 17:46:37 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session opened for u>
Feb 21 18:01:01 host.lab.example.com CROND[24468]: (root) CMD (run-parts /etc/ cron.hourly)
Feb 21 18:01:01 host.lab.example.com run-parts[24471]: (/etc/cron.hourly) starting 0anacron
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
lines 1464-1487/1487 (END) q

The journalctl command highlights important log messages: messages at notice or warning priority are in bold text while messages at the error priority or higher are in red text. The key to successfully using the journal for troubleshooting and auditing is to limit journal searches to show only relevant output.

By default, journalctl -n shows the last 10 log entries. You can adjust this with an optional argument that speciﬁes how many log entries to display. For the last ﬁve log entries, run the following journalctl command:

[root@host ~]# journalctl -n 5
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:01:01 +07.
-
...output omitted...
Feb 21 17:46:37 host.lab.example.com systemd-logind[708]: New session 20 of user root.
Feb 21 17:46:37 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session opened for u>
Feb 21 18:01:01 host.lab.example.com CROND[24468]: (root) CMD (run-parts /etc/ cron.hourly)
Feb 21 18:01:01 host.lab.example.com run-parts[24471]: (/etc/cron.hourly) starting 0anacron
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
lines 1-6/6 (END) q

Similar to the tail -f command, the journalctl -f command outputs the last 10 lines of the system journal and continues to output new journal entries as they get written to the journal. To exit the journalctl -f process, use the Ctrl+C key combination.

[root@host ~]# journalctl -f
-- Logs begin at Wed 2019-02-20 16:01:17 +07. -
...output omitted...
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
Feb 21 18:22:42 host.lab.example.com sshd[24437]: Received disconnect from 172.25.250.250 port 48710:11: disconnected by user
Feb 21 18:22:42 host.lab.example.com sshd[24437]: Disconnected from user root 172.25.250.250 port 48710
Feb 21 18:22:42 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session closed for user root
Feb 21 18:22:42 host.lab.example.com systemd-logind[708]: Session 20 logged out.
Waiting for processes to exit.
Feb 21 18:22:42 host.lab.example.com systemd-logind[708]: Removed session 20.
Feb 21 18:22:43 host.lab.example.com sshd[24499]: Accepted publickey for root from 172.25.250.250 port 48714 ssh2:
RSA SHA256:1UGybTe52L2jzEJa1HLVKn9QUCKrTv3ZzxnMJol1Fro
Feb 21 18:22:44 host.lab.example.com systemd-logind[708]: New session 21 of user root.
Feb 21 18:22:44 host.lab.example.com systemd[1]: Started Session 21 of user root.
Feb 21 18:22:44 host.lab.example.com sshd[24499]: pam_unix(sshd:session): session opened for user root by (uid=0)
^C
[root@host ~]#

To help troubleshoot problems, you might want to ﬁlter the output of the journal based on the priority of the journal entries. The journalctl -p takes either the name or the number of a priority level and shows the journal entries for entries at that priority and above. The journalctl command understands the debug, info, notice, warning, err, crit, alert, and emerg priority levels.

Run the following journalctl command to list journal entries at the err priority or higher:

[root@host ~]# journalctl -p err
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:01:01 +07.
-
..output omitted...
Feb 20 16:01:17 host.lab.example.com kernel: Detected CPU family 6 model 13 stepping 3
Feb 20 16:01:17 host.lab.example.com kernel: Warning: Intel Processor - this hardware has not undergone testing by Red Hat and might not be certif>
Feb 20 16:01:20 host.lab.example.com smartd[669]: DEVICESCAN failed: glob(3)
aborted matching pattern /dev/discs/disc*
Feb 20 16:01:20 host.lab.example.com smartd[669]: In the system's table of devices
NO devices found to scan
lines 1-5/5 (END) q

When looking for speciﬁc events, you can limit the output to a speciﬁc time frame. The journalctl command has two options to limit the output to a speciﬁc time range, the –since and –until options. Both options take a time argument in the format “YYYY-MM-DD hh:mm:ss” (the double-quotes are required to preserve the space in the option). If the date is omitted, the command assumes the current day, and if the time is omitted, the command assumes the whole day starting at 00:00:00. Both options take yesterday, today, and tomorrow as valid arguments in addition to the date and time ﬁeld.

Run the following journalctl command to list all journal entries from today’s records:

[root@host ~]# journalctl --since today
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:31:14 +07.
 -
...output omitted...
Feb 21 18:22:44 host.lab.example.com systemd-logind[708]: New session 21 of user root.
Feb 21 18:22:44 host.lab.example.com systemd[1]: Started Session 21 of user root.
Feb 21 18:22:44 host.lab.example.com sshd[24499]: pam_unix(sshd:session):
session opened for user root by (uid=0) Feb 21 18:31:13 host.lab.example.com systemd[1]: Starting dnf makecache...
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Red Hat Enterprise Linux 8.0
AppStream (dvd)    637 kB/s | 2.8 kB     00:00
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Red Hat Enterprise Linux 8.0
BaseOS (dvd)       795 kB/s | 2.7 kB     00:00
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Metadata cache created.
Feb 21 18:31:14 host.lab.example.com systemd[1]: Started dnf makecache. lines 533-569/569 (END) q

Run the following journalctl command to list all journal entries ranging from 2019-02-10 20:30:00 to 2019-02-13 12:00:00.

[root@host ~]# journalctl --since "2014-02-10 20:30:00" --until "2014-02-13 12:00:00"
...output omitted...

You can also specify all entries since a time relative to the present. For example, to specify all entries in the last hour, you can use the following command:

[root@host ~]# journalctl --since "-1 hour"
...output omitted...

Note: You can use other, more sophisticated time speciﬁcations with the –since and -until options. For some examples, see the systemd.time(7) man page.

In addition to the visible content of the journal, there are ﬁelds attached to the log entries that can only be seen when verbose output is turned on. Any displayed extra ﬁeld can be used to ﬁlter the output of a journal query. This is useful to reduce the output of complex searches for certain events in the journal.

[root@host ~]# journalctl -o verbose
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:31:14 +07.
--
...output omitted...
Thu 2019-02-21 18:31:14.509128 +07...
PRIORITY=6
    _BOOT_ID=4409bbf54680496d94e090de9e4a9e23
    _MACHINE_ID=73ab164e278e48be9bf80e80714a8cd5
    SYSLOG_FACILITY=3
    SYSLOG_IDENTIFIER=system
    _UID=0
    _GID=0
    CODE_FILE=../src/core/job.c
    CODE_LINE=826
    CODE_FUNC=job_log_status_message
    JOB_TYPE=start
    JOB_RESULT=done
    MESSAGE_ID=39f53479d3a045ac8e11786248231fbf
    _TRANSPORT=journal
    _PID=1
    _COMM=system
    _EXE=/usr/lib/systemd/system
    _CMDLINE=/usr/lib/systemd/system --switched-root --system --deserialize 18
    _CAP_EFFECTIVE=3fffffffff
    _SELINUX_CONTEXT=system_u:system_r:init_t:s0
    _SYSTEMD_CGROUP=/init.scope
    _SYSTEMD_UNIT=init.scope
    _SYSTEMD_SLICE=-.slice
    UNIT=dnf-makecache.service
    MESSAGE=Started dnf makecache.
    _HOSTNAME=host.lab.example.com
    INVOCATION_ID=d6f90184663f4309835a3e8ab647cb0e
    _SOURCE_REALTIME_TIMESTAMP=1550748674509128
lines 32239-32275/32275 (END) q

The following list gives the common ﬁelds of the system journal that can be used to search for lines relevant to a particular process or event.

_COMM The name of the command
_EXE The path to the executable for the process
_PID The PID of the process
_UID The UID of the user running the process
_SYSTEMD_UNIT The systemd unit that started the process

More than one of the system journal ﬁelds can be combined to form a granular search query with the journalctl command. For example, the following journalctl command shows all journal entries related to the sshd.service systemd unit from a process with PID 1182.

[root@host ~]# journalctl _SYSTEMD_UNIT=sshd.service _PID=1182
Apr 03 19:34:27 host.lab.example.com sshd[1182]: Accepted password for root
from ::1 port 52778 ssh2
Apr 03 19:34:28 host.lab.example.com sshd[1182]: pam_unix(sshd:session): session opened for user root by (uid=0)
...output omitted...

Note: For a list of commonly used journal ﬁelds, consult the systemd.journalfields(7) man page.

Beginners Guide to CentOS/RHEL logging (systemd-journald and rsyslog) architecture
Overview of Syslog Priorities, rsyslog rules, log rotation and basic troubleshooting using syslog
How to use journalctl to view log messages in journal for system troubleshooting
How to configure the system journal to preserve the record of events when a server is rebooted (CentOS/RHEL)

How to use journalctl to view log messages in journal for system troubleshooting

Finding Events

Latest Posts