Monit makes Mongrel play nice!

By Ilya Grigorik on November 07, 2006

It's 8am am on a Monday morning, having finished my breakfast I park my enormous coffee mug on the desk, fire up Firefox and point it to my paradigm-shifting Rails application. I want to check the overnight activity, but instead Apache is shouting back at me: "Service is temporarily unavailable"! What happened!? (Insert a sarcastic Monday morning joke here)

Well, after a quick inspection I realize that Apache is in fact alive and doing well, but my mongrel cluster is dead! The error message is reported by mod_proxy since it is unable to locate any mongrel severs on the specified ports. Not good. After restarting the Mongrel cluster I start digging through the logs to see what happened, and sure enough I find that all Mongrel servers Seg. Faulted in one of the libraries (Ferret, if you're curious). On further inspection, I also find that I'm also not the only one with such problems. Mongrel on its own is fairly stable, but in practice, as we all know, bad things happen and we need to have a way to recover from them. And that is where Monit comes in:

Monit is a utility for managing and monitoring, processes, files, directories and devices on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Nice thing is, we already have a cluster of mongrels, so the users will not notice when one of the servers dies as long as some other processes are running, all we have to guarantee is that at least one server is available to meet the minimum requirements of a functioning web-site. So let's get to it, in your shell:

$ wget http://www.tildeslash.com/monit/dist/monit-4.8.1.tar.gz
$ tar zxvf monit
$ cd monit
$ ./configure
$ make && make install

Now you have Monit installed on your system, in my case the executable was at /usr/local/bin/monit. Let's configure it to make Mongrel play nice! In your installation directory (where you compiled Monit from) you can find monitrc, a well documented sample configuration file. One feature that makes Monit stand out is its configuration syntax, it is extremely easy to follow and offers a lot of neat features. Take a look at my config:

set daemon 60
set mailserver localhost
set mail-format { from: monit@yourserver.com }
set alert myemail@domain.com

set httpd port 2812 and
    use address localhost  # only accept connection from localhost
    allow localhost        # allow localhost to connect to the server and

##### mongrel 8010 #####
check process mongrel-8010 with pidfile /home/user/rails/current/log/mongrel.8010.pid
    start program = "/usr/local/bin/ruby /usr/local/bin/mongrel_rails start -d -e production -p 8010 -a 127.0.0.1 -P /home/user/current/log/mongrel.8010.pid -c /home/user/rails/current"
    stop program  = "/usr/local/bin/ruby /usr/local/bin/mongrel_rails stop -P /home/user/current/log/mongrel.8010.pid"

    if totalmem is greater than 60.0 MB for 5 cycles then restart       # eating up memory?
    if cpu is greater than 50% for 2 cycles then alert                  # send an email to admin
    if cpu is greater than 80% for 3 cycles then restart                # hung process?
    if loadavg(5min) greater than 10 for 8 cycles then restart          # bad, bad, bad
    if 3 restarts within 5 cycles then timeout                         # something is wrong, call the sys-admin

    if failed port 8010 protocol http                   # check for response
        with timeout 10 seconds
        then restart
    group mongrel

Update: apply this mongrel_rails patch to force mongrel to delete stale pid files.

Let's break it down. Daemon directive tells Monit to check the services every 60 seconds (keep it reasonable!) Next 3 lines configure the email notifications - should be self explanatory, can be omitted. Monit also comes with its own lightweight web-server which allows you to monitor the status of the services and the system as a whole, you don't have to run it, but it's highly recommended. As you can see, I only allowed access from localhost for security reasons. How do you access it you say? Try lynx localhost:2812 in your console, works great! Now, let's fix our Mongrel! Important things to note:

• You need name your process (mongrel-8010) and specify the PID (Process ID) file location
• Specify a startup and shutdown command
• Specify your rules and assign a process group (mongrel)
• Repeat for all other mongrel processes

Save your configuration file, and we are ready to roll: /usr/local/bin/monit -c /path/to/monitrc

Monit should now be running, fire up lynx and see for yourself, you can select any of your mongrel servers and drill down to see more information on its status, stop it, start it, and restart it. At this point you have probably realized that we are manually managing each mongrel, which means we forfeit the mongrel_cluster functionality. And that is partially true, theoretically you could monitor the 'cluster', but restarting the entire pack of servers on a single failure is not recommended. However, we can make use of the group mongrel directive in Monit:

/usr/local/bin/monit -g mongrel [stop|start|restart] all

In our config we assigned every mongrel process to the 'mongrel' group, and here we can control them all with a single command. There is plenty of other handy features, check out the documentation for more! You can also setup Monit to check your Lighty, Apache, Mail, SQL, even your toaster! (if it has a network card) For some good examples click here or here.

Ilya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.