marathon

How to Keep Your Websites Running


If your web server is on a wimpy VPS (such as the one I had with Arvixe), you may need to monitor your blog to ensure that it is always up. Sure, there are such monitoring services; JetPack, for instance, that can send you an email. Then you will have to log in and restart your web server. The whole process is cumbersome and prone to unnecessary delays. This article outlines an automatic procedure to keep your websites running.

In order to automate the whole thing, all you have to do is to set up a scheduled task to do whatever it is that you do to check and restart your web server. Here is what the whole thing entails:

  1. Check if your web server is up (by ensuring that a test page loads within a certain time, for instance)
  2. If not, kill the web server.
  3. Ensure that the web server is really dead. If not, go back to 2.
  4. If it doesn’t die within a certain span of time, force kill i.t
  5. Relaunch the web server.
  6. Relaunch any other servers that you need to.

That is the theory. How do you do it in practice? The following scripts will help, if you are on a Linux VPS or dedicated server, and if you have root access to your server.

Here is how I did it on Arvixe. (I don’t have to do this any more, now that my sites are on a good VPS provided by InMotion. But it doesn’t hurt to have this auto-check running.) This script and procedure will work for CentOS, running a reverse-proxied Apache with an nginx front-end. If you don’t have a reverse proxy, don’t worry. The script becomes simpler.

A word of warning before we proceed though. This script and procedure are meant for people who are comfortable working as root on their linux machines. If you make a mistake as root, it can seriously mess up your system and prevent your access to your web server, VPS or dedicated server so thoroughly that you may have a hard time recovering. Please don’t do this unless you understand how the script works, and you are proficient enough to modify it to suit your purpose. You have been warned.

With that warning out of the way, here is the first script you need. Call it checkApache.sh.

#!/bin/sh

# Apache port. If unsure, set it to its default 80
port=80

# The URL which is polled to see if your webserver is running
url=your-site.com:$port/phpinfo.php

# Your emal where a notification will be sent, if apache is restarted.
mail=you@your-site.com

# Max load (in percent) on your server before apache is restarted automatically
limit=1000

# Time to wait for a response from your server before it is restarted
timeout=60

# Time allotted for the server to respond
maxtime=90

######### End of Configurations ###############

bounceApache() {
    /root/bounceApache.sh $3
    cat /var/log/checkApache.log | mail -s "$1" $2
    exit
}

w=`w | head -1| awk -F, '{print $NF*100}' | tr -d ' '`
echo "`date`: Load average is $w%"
if [ $w -gt $limit ]
then
    echo "`date`: High load (load =$w, limit = $limit). Bouncing Apache now."
    bounceApache "Apache Bounced on Arvixe: Heavy load ($w%)" $mail $port
fi
echo "`date`: Checking the server with Curl..."
if (curl --connect-timeout $timeout -m $maxtime $url > /dev/null 2>&1)
then
    echo "`date`: Curl works fine on $url."
else
    echo "`date`: Curl fails on $url... Bouncing Apache now."
    bounceApache "Apache Bounced on Arvixe: Curl fails" $mail $port
fi
chk=`netstat -an | grep -c :$port`
if [ "$chk" = "0" ]
then
    echo "`date`: Apache is down (netstat gives 0 connections to $port). Starting it...";
    /usr/local/apache/bin/apachectl start
    echo "`date`: Done"
    exit
fi
echo "`date`: Apache is running fine!";

As you can see, this script uses another script to safely (and surely) restart apache. Call this one bounceApache.sh, to be found in the home directory of root, which would be /root on CentOS.

#!/bin/sh

# Uncomment the next line to run this script in non-execute mode
# echo=echo

# Time allotted for apache processes to exit gracefully
timeout=120

######### End of Configurations ###############

# Apache port. First argument. Defaults to 80.
port=${1:-80}

chk=`netstat -an | grep -c :$port`
if [ "$chk" = "0" ]
then
   echo "`date`: Apache is down, starting it...";
   $echo /usr/local/apache/bin/apachectl start
   echo "`date`: Done"
   exit
fi

runtime=0
echo "`date`: Killing Apache..."
$echo /usr/local/apache/bin/apachectl stop
echo "`date`: Checking if it is dead..."
while [ `netstat -an | grep -c :$port` -gt 0 ]
do
  echo "`date`: Found `netstat -an | grep -c :$port` httpd proceses. Sleeping for 5 secs... [total = $runtime]"
  sleep 5
  runtime=$[$runtime + 5]
  if [ $runtime -gt $timeout ]
  then
      echo "`date`: Timing out apachectl stop. Will try killall httpd"
      $echo killall httpd
      sleep 30
      break
  fi
done

echo "`date`: Restarting..."
$echo /usr/local/apache/bin/apachectl restart

# If you have nginx as a reverse-proxy frontend...

if [[ $port != 80 ]]
then
    echo "`date`: Restarting NGINX.."
    $echo /sbin/service nginx restart || $echo /sbin/service nginx restart
fi
echo "`date`: Done"

Copy the scripts and create the files with the right names in /root and you are almost there. The next thing to do is to set up a cron job to execute it on a regular basis. The command to do that would be:

crontab -e

In the editor window, enter the following to run this script every five minutes. Or a frequency of your choosing.

*/5 * * * * /root/checkApache.sh > /var/log/checkApache.log 2>&1

Note that you will have to set the scripts as executable.

chmod a+x bounceApache.sh checkApache.sh