Free up disk space with efficient log file management

In a previous post, we looked at identifying disk usage and examined how to reclaim some of this space. One of the most common causes of low disk space (particularly the /var partition) is large log files.
Log files are utilised by many elements of the LAMP server, and these range tremendously in what they cover. Most web developers are likely to be exclusively concerned with the web logs – however, when problems occur, often the less familiar system logs become essential in tracking down the cause of the problem.

If not managed properly, log files can grow out of control and eat up excessive amounts of disk space. For this reason a tool exists in Linux called Logrotate. Logrotate allows you to archive logs after a they reach a certain size or age, and you can define how many of these are kept.

Because Linux has many of its own logs, and because Logrotate is the preferred way of handling these, you will probably find that this tool is already installed. To check whether or not your system includes logrotate run the following command:

logrotate --help

If it is installed, this should output something like the following:

root@admin@14:43 ; logrotate -?
Usage: logrotate [OPTION...] 
   -d, --debug               Don't do anything, just test (implies -v)
   -f, --force               Force file rotation
   -m, --mail=command        Command to send mail (instead of 
`/usr/bin/mail')
   -s, --state=statefile     Path of state file
   -v, --verbose             Display messages during rotation

Help options:
   -?, --help                Show this help message
       --usage               Display brief usage message

If it's not installed, you can install it using your distribution's software package manager (such as yum, or apt). If it is unavailable in your distribution, you might need to build it from source.

NB: It is worth bearing in mind that services usually require restarting if their log files are changed, e.g., if you change the Apache log files, Apache will need to be restarted. Some services are built with this in mind and provide you with a less intrusive way to request that it refreshes its file handles - for this purpose the SIGHUP signal is generally used. Later on we'll cover how to do this with Logrotate.

Logrotate works via cron, which is a commonly implemented service across Unix flavours providing a framework for programs to be run at specific times automatically. It's a job scheduler essentially, and through it Logrotate is usually set to run daily in the early hours of the morning, when the server is usually quieter and under less load. This can be changed as required, just like any other cron job.

Logrotate essentially has a single configuration, however this is usually split across several files. Base directives (to be inherited by specific configurations) are normally stored in the main configuration file (/etc/logrotate.conf) and the individual configuration files for specific groups of logs (httpd, exim, etc) are stored inside files in the directory /etc/logrotate.d (which is included by the main configuration file).

Usually the base directives will define something along the lines of weekly rotation with 4 old rotations to be compressed and kept. The specific configurations can override these values, but only inside the scope of the files that configuration block is being applied to.

Lets review a simple configuration for some log files:

/var/log/blue.log /var/log/red.log {
     size 100k
     rotate 2
     compress
     missingok
     notifempty
     copytruncate
}

As you can see, this rule only applies to the files /var/log/blue.log and /var/log/red.log. Also it overrides any base directives concerning retention periods and rotation rate, setting them respectively to 2 and whenever the file reaches 100 kilobytes. This size declaration is used where you could alternatively define daily, weekly or monthly.

In addition to this compression is enabled for rotations, meaning that old copies are compressed to save space. Next the missingok declaration means that if either of the log files do not exist, rather than reporting an error it will silently continue.
The notifempty directive tells Logrotate not to rotate a file if it's empty, usually this would seem like the most sane thing to do, however there might be times when you may want even empty ones rotated to preserve the numbering across order across specific logs.

Finally the copytruncate declaration tells Logrotate not replace the log file, but to instead copy the contents of it into a new file, and then truncate the original one - this has the benefit of not needing to tell the service to reload in order to update its file handle to the new log file, although this method has the drawback of losing whatever is written between the copy and the truncation.

Now let's look at another example, in this one I'll attempt to demonstrate some more commonly used options:

/var/log/silver.log /var/log/gold.log {
     rotate 3
     weekly
     ifempty
     nocompress
     missingok
     create 644 second applications
     postrotate
         kill -HUP $(pidof secondApplicationServer)
     endscript
}

In this example we retain 3 old versions and rotate on a weekly basis - even if empty. No compression is applied to old logs. If either of the log files are missing it'll simply carry on. New log files are created (the old ones are moved rather than copied), and these new files are created owned by user 'second' and group 'applications'. Finally, after rotation is performed the command 'kill -HUP $(pidof secondApplicationServer)' is run. Sometimes you'll need to send a signal to an application to request that it refreshes the filehandles that it has active – specifically for log files.

When you begin writing your own Logrotate rules it would useful to test them before they run. To achieve this the Logrotate binary provides you with two switches, -d and -v that facilitate this. The -d switch turns on the debugging mode, while operating in this mode logrotate will never make any changes to your files. Secondly, the -v switch turns on verbose output, this'll report to you exactly what logrotate would be doing (even if it doesn't due to due to the -d switch).

Let's see what Logrotate will do with the first of the two examples:

root@delli:~# logrotate -dv /etc/logrotate.d/first
reading config file /etc/logrotate.d/first
reading config info for /var/log/blue.log /var/log/red.log

Handling 1 logs

rotating pattern: /var/log/blue.log /var/log/red.log  102400 bytes (2 
rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/blue.log
   log /var/log/blue.log does not exist -- skipping
considering log /var/log/red.log
   log needs rotating
rotating log /var/log/red.log, log->rotateCount is 2
renaming /var/log/red.log.2 to /var/log/red.log.3 (rotatecount 2, 
logstart 1, i 2),
renaming /var/log/red.log.1 to /var/log/red.log.2 (rotatecount 2, 
logstart 1, i 1),
renaming /var/log/red.log.0 to /var/log/red.log.1 (rotatecount 2, 
logstart 1, i 0),
copying /var/log/red.log to /var/log/red.log.1
truncating /var/log/red.log
removing old log /var/log/red.log.3

Here you can see that Logrotate skips blue.log because it doesn't exist, and continues to process red.log. After this the file red.log is considered for rotation immediately, however if the rule for this file demanded that it was rotated daily, weekly or monthly it wouldn't be considered as logrotate itself keeps track of age from the first time that it encounters the file. Usually this data is maintained in the file /var/lib/logrotate.status or /var/lib/logrotate/status - however these files are used internally by logrotate and do not require manual modification.

Logrotate is staple tool in Linux system administration and as shown it offers a lot of control over managing your log files. In some circumstances other tools may be useful, for example Tomcat provides its own mechanisms to handle log files - eliminating the need to reload the service or risk losing data in a copytruncate.