Make syslog work harder, and spend less time looking through huge log files.
The default syslog installation on many distributions doesn’t do a very good job of filtering classes of information into separate files. If you see a jumble of messages from Sendmail, sudo, BIND, and other system services in /var/log/messages, then you should probably review your /etc/syslog.conf.
There are a number of facilities and priorities that syslog can filter on. These facilities include auth, auth-priv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, and local0 through local7. In addition, each facility can have one of eight priorities: debug, info, notice, warning, err, crit, alert, and emerg.
Note that applications decide for themselves at what facility and priority to log (and the best apps let you choose), so they may not be logged as you expect. Here’s a sample /etc/syslog.conf that attempts to shuffle around what gets logged where:
auth.warning /var/log/auth
mail.err /var/log/maillog
kern.* /var/log/kernel
cron.crit /var/log/cron
*.err;mail.none /var/log/syslog
*.info;auth.none;mail.none /var/log/messages
#*.=debug /var/log/debug
local0.info /var/log/cluster
local1.err /var/log/spamerica
All of the lines in this example will log the specified priority (or higher) to the respective file. The special priority none tells syslog not to bother logging the specified facility at all. The local0 through local7 facilities are supplied for use with your own programs, however you see fit. For example, the /var/log/spamerica file fills with local1.err (or higher) messages that are generated by our spam processing job. It’s nice to have those messages separate from the standard mail delivery log (which is in /var/log/maillog).
The commented *.=debug line is useful when debugging daemonized services. It tells syslog to specifically log only debug priority messages of any facility, and generally shouldn’t be running (unless you don’t mind filling your disks with debug logs). Another approach is to log debug information to a fifo. This way, debug logs take up no space, but they will disappear unless a process is watching it. To log to a fifo, first create it in the filesystem:
# mkfifo -m 0664 /var/log/debug
Then amend the debug line in syslog.conf to include a |, like this:
*.=debug |/var/log/debug
Now debug information is constantly logged to the fifo and can be viewed with a command like less -f /var/log/debug. A fifo is also handy if you want a process to constantly watch all system messages and perhaps notify you via email about a critical system message. Try making a fifo called /var/log/monitor, and add a rule like this to your syslog.conf:
*.* |/var/log/monitor
Now every message (at every priority) is passed to the /var/log/monitor fifo, and any process watching it can react accordingly, all without taking up any disk space.
Mark Who?
Do you notice a bunch of lines like this in /var/log/messages?
Dec 29 18:33:35 catlin — MARK —
Dec 29 18:53:35 catlin — MARK —
Dec 29 19:13:35 catlin — MARK —
Dec 29 19:33:35 catlin — MARK —
Dec 29 19:53:35 catlin — MARK —
Dec 29 20:13:35 catlin — MARK —
Dec 29 20:33:35 catlin — MARK —
Dec 29 20:53:35 catlin — MARK —
Dec 29 21:13:35 catlin — MARK —
These are generated by the mark functionality of syslog, as a way of “touching base” with the system, so that you can (theoretically) tell if syslog has unexpectedly died. Most times, this only serves to fill your log files, and unless you are having problems with syslog, you probably don’t need it. To turn this off, pass the -m 0 switch to syslogd (after first killing any running syslogd), like this:
# killall syslogd; /usr/sbin/syslogd -m 0
If all of this fiddling about with facilities and priorities strikes you as arcane Unix speak, you’re not alone. These examples are provided for systems that include the default (and venerable) syslogd daemon. If you have the opportunity to install a new syslogd, you will likely want to look into syslog-ng. This new implementation of syslogd allows much more flexible filtering and a slew of new features. We take a look at some of what is possible with syslog-ng in [Hack #59] .