Overview of Syslog Priorities, rsyslog rules, log rotation and basic troubleshooting using syslog
Logging events to the system
Many programs use the syslog protocol to log events to the system. Each log message is categorized by a facility (the type of message) and a priority (the severity of the message). Available facilities are documented in the rsyslog.conf(5) man page.
The following table lists the standard eight syslog priorities from highest to lowest.
|0||emerg||System is unusable|
|1||alert||Action must be taken immediately|
|3||err||Non-critical error condition|
|5||notice||Normal but significant event|
The rsyslog service uses the facility and priority of log messages to determine how to handle them. This is conﬁgured by rules in the /etc/rsyslog.conf ﬁle and any ﬁle in the /etc/rsyslog.d directory that has a ﬁle name extension of .conf. Software packages can easily add rules by installing an appropriate ﬁle in the /etc/rsyslog.d directory.
Each rule that controls how to sort syslog messages is a line in one of the conﬁguration ﬁles. The left side of each line indicates the facility and severity of the syslog messages the rule matches. The right side of each line indicates what ﬁle to save the log message in (or where else to deliver the message). An asterisk (*) is a wildcard that matches all values.
For example, the following line would record messages sent to the authpriv facility at any priority to the ﬁle /var/log/secure:
Log messages sometimes match more than one rule in rsyslog.conf. In such cases, one message is stored in more than one log ﬁle. To limit messages stored, the keyword none in the priority ﬁeld indicates that no messages for the indicated facility should be stored in the given ﬁle. Instead of logging syslog messages to a ﬁle, they can also be printed to the terminals of all logged-in users. The rsyslog.conf ﬁle has a setting to print all the syslog messages with the emerg priority to the terminals of all logged-in users.
Sample Rules of rsyslog
#### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log
Log File Rotation
The logrotate tool rotates log ﬁles to keep them from taking up too much space in the ﬁle system containing the /var/log directory. When a log ﬁle is rotated, it is renamed with an extension indicating the date it was rotated. For example, the old /var/log/messages ﬁle may become /var/log/messages-20190130 if it is rotated on 2019-01-30. Once the old log ﬁle is rotated, a new log ﬁle is created and the service that writes to it is notiﬁed.
After a certain number of rotations, typically after four weeks, the oldest log ﬁle is discarded to free disk space. A scheduled job runs the logrotate program daily to see if any logs need to be rotated. Most log ﬁles are rotated weekly, but logrotate rotates some faster, or slower, or when they reach a certain size. For more information, see the logrotate(8) man page.
Analyzing a syslog entry
Log messages start with the oldest message on top and the newest message at the end of the log ﬁle. The rsyslog service uses a standard format while recording entries in log ﬁles. The following example explains the anatomy of a log message in the /var/log/secure log ﬁle.
- The time stamp when the log entry was recorded
- The host from which the log message was sent
- The program or process name and PID number that sent the log message
- The actual message sent
Monitoring one or more log ﬁles for events is helpful to reproduce problems and issues. The tail -f /path/to/file command outputs the last 10 lines of the ﬁle speciﬁed and continues to output new lines in the ﬁle as they get written.
For example, to monitor for failed login attempts, run the tail command in one terminal and then in another terminal, run the ssh command as the root user while a user tries to log in to the system. In the ﬁrst terminal, run the following tail command:
[root@host ~]# tail -f /var/log/secure
In the second terminal, run the following ssh command:
[root@host ~]# ssh root@localhost root@localhost's password: mypass ...output omitted... [root@host ~]#
Return to the ﬁrst terminal and view the logs.
...output omitted... Feb 10 09:01:13 host sshd: Accepted password for root from 172.25.254.254 port 56801 ssh2 Feb 10 09:01:13 host sshd: pam_unix(sshd:session): session opened for user root by (uid=0)
Sending syslog messages manually
The logger command can send messages to the rsyslog service. By default, it sends the message to the user facility with the notice priority (user.notice) unless speciﬁed otherwise with the -p option. It is useful to test any change to the rsyslog service conﬁguration.
To send a message to the rsyslog service that gets recorded in the /var/log/boot.log log ﬁle, execute the following logger command:
[root@host ~]# logger -p local7.notice "Log entry created on host"