Unix Signals for Live Debugging

By on July 22, 2008

It's unit tested, the integration and regressions tests are all green, you even added a set of performance benchmarks, and yet after a couple of hours of production use the process is falling over - we've all been there before. Good instrumentation in these situations is worth its weight in gold. If you're lucky, you may be able to use DTrace, if you're adventurous you may try to attach with GDB, but what if you build this instrumentation into your app right from the start?

System signals for easy debugging

Any UNIX process can respond to a collection of system signals, and what's even better: we can safely define a subset (USR1, and USR2) to invoke arbitrary logic within our process. For example, let's create a worker process which on receipt of a USR1 signal will toggle debug mode on demand:

require 'rubygems'
require 'log4r'

# initialize the logger
@log = Log4r::Logger.new 'log'
@log.outputters = Log4r::StdoutOutputter.new 'console'
@log.level = Log4r::INFO

worker = Thread.new do
  while (true) do
    @log.info 'Hello!'
    @log.debug 'Debug!'
    sleep(1)
  end
end

# put worker into debug mode via USR1 signal
trap("USR1") {
 @log.level = @log.level == Log4r::INFO ? Log4r::DEBUG : Log4r::INFO
}

# cleanly shutdown the thread on Ctrl-C / kill signal
trap("INT") { worker.terminate }

# run the worker
worker.join
debug-mode.rb - Ruby file

In the example above Log4r provides us with a convenient logging hierarchy: DEBUG < INFO < WARN < ERROR < FATAL. By sending our process a USR1 signal (kill -s USR1 pid), we toggle between the INFO and DEBUG levels, allowing ourselves the luxury of performing live debugging without incurring the cost of redundantly logging debug information when we don't need it. Now you have the tools to perform that open-heart surgery.

INFO log: Hello! **< process is started**INFO log: Hello!INFO log: Hello! **< process receives USR1**DEBUG log: Debug! INFO log: Hello!DEBUG log: Debug! **< process receives USR1**INFO log: Hello!INFO log: Hello!

For bonus points, check what signals your favorite UNIX processes respond to. For example, did you know that kill -s USR1 mongrel_pid will turn on debug mode on any Mongrel process? Make use of system signals, they're there for reason!

Ilya GrigorikIlya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.