Watching your log files

It's a good idea to check your server's logs fairly regularly, preferably daily, for signs of intrusion attempts or system failures. However, this can be a long-winded process, and it's easy to forget to do this every day. One way of simplifying checking your logs (and helping to remember to check them), is to use Logwatch.

Logwatch is a popular log monitoring system, that can be set to email you a daily summary of the logs. It's written in Perl and, if not already installed, can easily installed using your distro's package management system:

For Red Hat/CentOS systems: yum install logwatch

For Debian/Ubuntu systems: apt-get install logwatch

Once installed, open the main configuration file using a text editor (e.g. vim) and have a look at some of the options:

vim /usr/share/logwatch/default.conf/logwatch.conf

There are three main options you might initially want to change:

1) LogDir = /var/log

All log files parsed are assumed to be relative to the path listed here. Usually, this won't need changing, but if your log files aren't in /var/log, you will need edit this value and add the correct path.

2) MailTo = root

By default, Logwatch sends it's reports to the root mailbox. If you want it emailed to you, add your email address here, e.g.

MailTo = admin@example.com

3) Detail = Low

This sets the level of information included in the report. Low is the default, but it can also be set to medium or high, if you need more information.

If you want to check if the detail option selected provides enough (or too much) information, you can run Logwatch from the command line, and output the report to the screen, to check what the report will contain, using the --print option, e.g.

logwatch --detail high --print

This will display the report on screen, with a high level of information. If you don't use the --detail option, it will generate the report using whatever level of reporting is set in logwatch.conf.

The default configuration assumes all log files are stored in one location - /var/log. If you have logs that are stored elsewhere, usually web logs, you will need to create a custom logfile filter. These custom filters are usually located in:

/etc/logwatch/conf/logfiles/

For example, let's assume you have a cPanel server, where the web logs are stored inside the folder /usr/local/apache/domlogs/, and you want the logs for your site example.com to be parsed by Logwatch. You would create a new file using vim:

vim /etc/logwatch/conf/logfiles/http.conf

And define the new log file location by adding this to the file:

Logfile =/usr/local/apache/domlogs/example.com

Save the new file, and then when Logwatch is next run, it will also check the example.com log file.

Note: Multiple Logfiles can be defined in this way, so it doesn't matter where your log files are located; as long as they are setup in a file within the logfiles directory, they will be read by Logwatch.

Finally, add a cronjon to run Logwatch daily. If you've setup everything correctly in logwatch.conf, you can run it without any options:

0 1 * * * /usr/sbin/logwatch

This will run Logwatch at 1am every morning, using whatever values have been setup in logwatch.conf.

If you haven't altered the conf file, you can add options to the cronjob, e.g.

0 1 * * * /usr/sbin/logwatch --detail high --mailto admin@example.com

This will run Logwatch at 1am every morning, generate a report using a high level of detail, and then email it to admin@example.com.

This is just a basic setup, and there are many more things you can do with Logwatch, but hopefully it's enough to get you started.

For further reading go to:

http://sourceforge.net/projects/logwatch/

http://linuxcommand.org/man_pages/logwatch8.html

Working Efficiently With Log Files

When dealing with log files, it's easy to get lost and end up spending ages looking through them. Here's a few tips to help you find what you're looking for faster.

Firstly, each entry in your log files will have a date stamp. You can use the following 'sed' command to find out the start and end dates of the log:

root@server [~]> sed -n -e '1p' -e '$p' /var/log/messages.3
Jul 4 04:02:14 170094 syslogd 1.4.1: restart.
Jul 11 04:01:42 170094 yum: net-tools-1.60-81.el5.x86_64: 100
root@server [~]>

By performing a directory listing these date stamps can be used to show you when a log file was last modified:

root@server [~]> ls -l /var/log/messages*
-rw------- 1 root root 2244927 Jul 30 15:20 /var/log/messages
-rw------- 1 root root 4663646 Jul 25 04:02 /var/log/messages.1
-rw------- 1 root root 7281102 Jul 18 04:00 /var/log/messages.2
-rw------- 1 root root 2362032 Jul 11 04:01 /var/log/messages.3
-rw------- 1 root root 3647411 Jul 4 04:01 /var/log/messages.4

Here we can see that the file messages.4 is almost guaranteed to go back to June 27. But when you encounter a single log file without any pattern of rotations it is often useful to grab the first few entries of it to determine when it begins. This is where the command 'head' comes in useful. By default it will take the first 10 lines from a file, or standard input if you're piping to it, otherwise you can set the number of lines like in the example below.

root@server [~]> head -5 /tmp/zabbix_agentd.log
5578:20100730:090202 zabbix_agentd started. ZABBIX 1.4.6.
5579:20100730:090202 zabbix_agentd collector started
5581:20100730:090202 zabbix_agentd listener started
5583:20100730:090202 zabbix_agentd listener started
5585:20100730:090202 zabbix_agentd listener started

Similar to head, the command 'tail' allows you to display the bottom lines from files. Tail also includes a switch that allows it to 'follow' a file as it is being written to. This is useful when you're troubleshooting a running service - such as the Apache while a PHP script is failing to work correctly - as it allows you to see the logging as it is generated. To use the follow feature simply add the -f switch when using tail, like in the command below:

root@server [~]> tail -f error.log

Sometimes it is useful to makes copies of sections of a log file when investigating a problem. Below is an example of how you would take the log entries from July 29 to July 30 from an Apache error log and copy them into another file:

root@server [~]> sed -n '/\[Thu Jul 29/,/\[Fri Jul 30/p' error.log > /tmp/errorLog.text

If you've ever had an obscene number of requests from the same IP or subnet hitting your machine you might be familiar with the necessity of extracting proof from the log files in order to submit an abuse claim to the IP provider. This is very straight forward and simply involves searching for the IP address, or the common part of the subnet, and directing the output to another file.

root@server [~]> egrep '10.0(\.[0-2]?[0-9]{1,2}){2}' access.log > /tmp/badAccessLog.excerpt

Furthermore, to simply search for an IP address use the following regular expression - this can be useful when searching for excessive connections. Here I've included a hat (^) at the beginning of the match, now it will only match possible IP address that appear at the start of a line - useful when browser revision version numbers can take the appearance of IP addresses:

root@server [~]> egrep -o '^([0-2]?[0-9]{1,2}\.){3}[0-2]?[0-9]{1,2}' access.log

The following command will provide you with an ordered list of the biggest hitters:

root@server [~]> egrep -o '^([0-2]?[0-9]{1,2}\.){3}[0-2]?[0-9]{1,2}' access.log | sort | uniq -c | sort -n
3 10.0.0.17
10 127.0.0.1
89 10.0.0.2

Sometimes you'll find that it's useful to reverse the output, which can be done by piping the output into 'tac'. Another use for tac is when you're grep'ing for something that you know will appear near the end of a very large log file. To speed up the retrieval of the appropriate entries, tac the file, then grep for the string you#re looking for - then tee direct it into a file (this way you'll know when you've caught it). Finally tac the output file to see the output in correct order. Here's an example.

root@server [~]> tac /var/log/maillog | grep 'me@lampguru' | tee /tmp/file.tmp
Jul 30 16:22:58 pulse sendmail[6446]: o6UFMtQu006444: to=, delay=00:00:03, xdelay=00:00:00, mailer=esmtp, pri=120225, relay=www.lampguru.co.uk. [213.52.229.86], dsn=2.0.0, stat=Sent (o6UFOq41015916 Message accepted for delivery)
Jul 30 16:00:15 pulse sendmail[5381]: o6UF0Cl4005378: to=, delay=00:00:03, xdelay=00:00:00, mailer=esmtp, pri=120225, relay=www.lampguru.co.uk. [213.52.229.86], dsn=2.0.0, stat=Sent (o6UF29m2010007 Message accepted for delivery)
root@server [~]> tac /tmp/file.tmp
Jul 30 16:00:15 pulse sendmail[5381]: o6UF0Cl4005378: to=, delay=00:00:03, xdelay=00:00:00, mailer=esmtp, pri=120225, relay=www.lampguru.co.uk. [213.52.229.86], dsn=2.0.0, stat=Sent (o6UF29m2010007 Message accepted for delivery)
Jul 30 16:22:58 pulse sendmail[6446]: o6UFMtQu006444: to=, delay=00:00:03, xdelay=00:00:00, mailer=esmtp, pri=120225, relay=www.lampguru.co.uk. [213.52.229.86], dsn=2.0.0, stat=Sent (o6UFOq41015916 Message accepted for delivery)

Finally, when using the 'less' command to view large files include the -n switch to turn off line numbering as this will improve performance.

NTP - updating server time

Network Time Protocol (NTP) is a protocol used to sychronise a computer's system clock. To ensure your server maintains the correct time, it is a good idea to set up cron job to update you time settings nightly using ntpdate.

To do this, first check to see whether NTP is installed. On a Red Hat system you can do this by running:

rpm -qa | grep ntpdate

On a Debian/Ubuntu system you would run:

dpkg -l | grep ntpdate

If it's not installed, you can install it using your distro's package manager. On Red Hat systems run:

yum install ntpdate

On Debian/Ubuntu systems run:

apt-get install ntpdate

You can then test it works by running:

ntpdate -q uk.pool.ntp.org

The -q switch runs ntpdate in query mode, causing it to report what changes would be made, but making no actual changes. The example shown runs the query against the UK NTP server pool. Different countries have different pools, e.g. de.pool.ntp.org for Germany, fr.pool.ntp.org for France, etc.

Next, we need to set up the cron job. First type:

crontab -e

This will open the crontab program in editing mode. Press 'i' to enter insert mode and type:

0 2 * * * /usr/sbin/ntpdate -s uk.pool.ntp.org

Then save and exit.

This will update the server time at 2am every morning, from NTP servers in the UK pool. The -s switch diverts any output to the normal system logs.

Note: remember to restart cron after editing crontab using: /etc/init.d/cron restart

ProFTP fixes

Here are a couple of fixes to some common ProFTP problems.

*** Cannot view .htaccess files

First, make sure your FTP client is actually set to view hidden files. If it is, and you still can't see any of your .htaccess files using an FTP connection, try the following:

Open the main ProFTP config file:

vi /etc/proftpd.conf

And look for the 'ListOptions' directive. If it doesn't exist, add the following:


ListOptions "-la"


Then save the changes and restart the proftpd daemon.

This will make all hidden files - i.e., file preceded by a '.' - viewable in long-format. The global tags apply the changes server-wide.

*** Timestamp shown on files is wrong

If the timestamp on the files appears to be an hour or so out, ProFTP might not be using the correct timezone. To change the time, open proftpd.conf and add the following:


SetEnv TZ BST


In this example, the SetnEnv directive sets ProFTP to use British Summer Time, but you can use any valid timezone abbreviation, as required, e.g. GMT, CEST, etc.

Updating SpamAssassin

SpamAssassin is a popular anti-spam program, that uses content-matching rules to scan email and filter out potential spam. For it to remain effective, it's best to make sure you're running the latest version, and that the rule definitions are updated regularly.

The current version, at time of writing, is 3.3.1 - assuming you're running an older version, let's look at how you would update it:

Note: This example assumes a 64-bit rpm based operating system (i.e. Red Hat or CentOS). If you're running Debian, or a 32-bit OS, please consult the official SpamAssassin website and adjust the instructions accordingly.

First, check which version of SpamAssassin is currently installed by running:

rpm -qa | grep spamassassin

Next, backup the main config file and directories. It's unlikely there will be any problems, but it's always best to make backups before upgrading - just in case...

Make a backup of local.cf and tar up a copy of the spamassassin directory by running:

cp -p /etc/mail/spamassassin/local.cf /etc/mail/spamassassin/BAK-local.cf
cd /usr/share
tar czvf BAK-spamassassin.tar.gz spamassassin/

Once this is done, download the latest tarball into the /usr/src directory:

cd /usr/src
wget http://www.mirrorservice.org/sites/ftp.apache.org//spamassassin/source/Mail-SpamAssassin-3.3.1.tar.gz

You can then use the tarball to build the installation rpms by running:

rpmbuild -tb Mail-SpamAssassin-3.3.1.tar.gz

This will create two rpm files in /usr/src/redhat/RPMS/x86_64/, which can be installed by running:

cd /usr/src/redhat/RPMS/x86_64/
rpm -Uvh perl-Mail-SpamAssassin-3.3.1-1.x86_64.rpm spamassassin-3.3.1-1.x86_64.rpm

Note: Remember to choose the correct path and rpm's for your build. The 64-bit ones are shown in the example.

The new version of SpamAssassin is now installed. Next, you need to update the rules by downloading and unpacking the latest ruleset:

cd /usr/share/spamassassin
wget http://mirror.ox.ac.uk/sites/rsync.apache.org//spamassassin/source/Mail-SpamAssassin-rules-3.3.1.r923114.tgz
tar -xzvf Mail-SpamAssassin-rules-3.3.1.r923114.tgz

This will unpack the new rules into the /usr/share/spamassassin directory.

Next run the SpamAssassin update tool - sa-update - to make sure everything is completely up-to-date. It's often useful to run it with the -D switch (debug mode), as this will display a list of everything that it's updating:

sa-update -D

The updates will be uploaded into the /usr/share/spamassassin/ directory. This directory includes a config file that contains 'Includes' for these new rules, which can be viewed by running:

less /var/lib/spamassassin//updates_spamassassin_org.cf

Next, stop sendmail, or MailScanner (if installed), and make sure that no processes are running before attempting a restart - using 'killall sendmail', if necessary. Then restart sendmail/MailScanner and tail your maillogs to make sure there are no errors.

To make sure the rules are update regularly, you can setup a cronjob to run sa-update each night. Here's an example of what you might add:

0 3 * * * /usr/bin/sa-update &> /dev/null; /etc/init.d/spamd restart

This will run sa-update at 3am every morning, piping any output produced to dev/null, and then restart spamd (the SpamAssassin daemon) to load the new rules.

For more information see: http://spamassassin.apache.org

Nginx and SSL

Nginx is an open source web server, and over the past few years it's started to become a popular alternative to the dominant Apache HTTP server.

Installing an SSL certificate on a server running Nginx isn't very difficult, particularly if you've ever installed one on Apache, but there are a few differences.

First, we need to make sure Nginx has been installed with SSL support enabled. To list the configuration options compiled into your installation of Nginx, run the command: nginx -V (You must use a capital 'V', otherwise it will only print the version number). Here's a sample output:

root@server:~$ nginx -V

nginx version: nginx/0.7.67

configure arguments: --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module –with-http_dav_module

From this, we can see that the –with-http_ssl_module option was applied when Nginx was configured, so SSL support is enabled.

Let's assume we already have a virtual host file set up for a non-secure version of the site example.co.uk. By default, Nginx uses a 'Debian style layout', so vhost files will be created in the sites-available folder, and then symlinked to the sites-enabled folder when they're ready to be made live. To edit the file, open it with your text editor of choice, e.g.

vim /etc/nginx/sites-available/example.co.uk

The vhost file should look something like this:

server {              listen   80;             server_name  www.example.co.uk;             rewrite ^/(.*) http://example.co.uk/$1 permanent;             }  server {              listen   80;             server_name example.co.uk;              access_log /home/example.co.uk/log/access.log;             error_log /home/example.co.uk/log/error.log;              location / {                          root   /home/example.co.uk/public_html/;                         index  index.html;                          }              }

Copy and paste a duplicate of each of these server modules and add them to the bottom of the vhost file, then change the port settings in this second set from 80 to 443. With these changes in place, Nginx will now listen for requests for example.co.uk on both port 80 and port 443.

We next need to enable SSL for the site and tell Nginx where to find the certificate. To do this, we need to add three lines between the listen and server_name directives:

            listen   443;      ssl on;   ssl_certificate /etc/ssl/certs/example.crt   ssl_certificate_key /etc/ssl/private/example.key              server_name example.co.uk;

Note: This assumes your certificate and key are stored in the /etc/ssl/ directory. Modify as necessary to point Nginx to the correct path.

Save the changes and restart Nginx – it's usually best to stop and start Nginx, rather than just issuing a restart, as the init script has been known to fail to apply changes when just restarted.

/etc/init.d/nginx stop

/etc/init.d/nginx start

And then test your secure site by going to:

https://www.example.co.uk

Chain certificates:

Chain certificates (also know as intermediate certificates) aren't defined in a separate file, as they are with Apache. You just need to add them to the bottom of the main certificate file. Assuming both files are in /etc/ssl/certs, you can append the main certificate file by running:

cat /etc/ssl/certs/intermediate.crt >> /etc/ssl/certs/example.crt

Make sure the intermediate certificate is below the main certificate, otherwise you will get a key mismatch error.

Further reading:

More information on the HttpSsl module can be found at:

http://wiki.nginx.org/NginxHttpSslModule

Installing Nginx on Red Hat/CentOS

Nginx is an open source web server, and over the past few years it's started to become a popular alternative to the dominant Apache HTTP server.

It is possible to install Nginx using the yum package manager, but it's a much better idea to install it from source.

You cannot install it using the standard repositories, so you'll need to add the EPEL (Extra Packages for Enterprise Linux) repository first. This is easily done using the latest installation rpm which can be found at:

http://fedoraproject.org/wiki/EPEL/FAQ#howtouse

Once EPEL is added to yum's repo list, you can install Nginx by simply running the command:

yum install nginx

The only problem being, the version available via EPEL is out of date. At time of writing, using yum will install version 0.6.36, which has been listed as a stable 'legacy' version since April 2009.

The current stable version is 0.7.67, which is thirteen versions up from the version available using yum, so it's always advisable to install it from source using the latest stable release.

The latest version can be downloaded from:

http://wiki.nginx.org/NginxInstall

This page also includes the basic build instructions. These are the standard three steps for building from source:

./configure
make
make install
(or sudo make install, if you're not root)

Before running configure, it's worth checking the list of compile-time options available. They're all listed at:

http://wiki.nginx.org/NginxInstallOptions

The modules installed as standard can be found listed at:

http://wiki.nginx.org/NginxModules

If there are any listed you do not want installed, you can use one of the '–without-*' arguments listed to exclude them.

Once Nginx is installed, you'll need to add an init script, as this won't be created when installing from source.

If you're feeling adventurous, you can try writing your own, otherwise you can use the standard version listed at:

http://wiki.nginx.org/RedHatNginxInitScript

This will respond to the familiar Apache style options, e.g. start, stop, restart, configtest, etc.