Asynchronous Database Access in Ruby

By Ilya Grigorik on September 05, 2008

These past few weeks have brought forward a few interesting developments in the Ruby database access layer. First, the guys at espace announced Neverblock: a Ruby 1.9 library which makes heavy use Fibers and non-blocking IO (RubyInside and InfoQ have the scoop). And shortly thereafter we saw the announcement of MySQLPlus (also a direct result of work around Neverblock): An enhanced MySQL database driver. With support for async operations and threaded database access. Nice!

Asynchronous panacea?

Does this mean we can turn ActiveRecord into a speed demon? Not quiet, or at least, not yet. Making our database calls asynchronous will allow us to handle many more parallel requests (which is a big win), without blocking the server, but it won't enhance the response time - the database is still the bottleneck. Furthermore, to take advantage of the non-blocking model, you will most likely have to adjust your application code (ack!).

Love it or hate it, ActiveRecord and its kin (Datamapper, Sequel, etc.) provide a great abstraction layer, which hides much of the database access logic and complexity. For these reasons, DBSlayer is a really interesting alternative, especially when used in conjunction with ActiveRecord or DataMapper adapters when it comes to asynchronous database access: language agnostic, speaks JSON, connection pooling, failover, plus many other goodies.

EM/MySQL - Ruby based asynchronous client

However, if you are interested in a Ruby alternative, keep an eye on Aman Gupta's recent project: em-mysql. An EventMachine wrapper around MySQLPlus, it is already showing very promising results:

The test consisted of 200 requests made at different concurrency levels against a single event-driven web-server. In turn, each request simulated a blocking database call (one second sleep), and then returned the results to the user. Hence, not surprisingly, 200 requests in serial order (concurrency of 1), took 200 seconds for DBSlayer, native MySQL driver, and EM/MySQL. However, as soon as we up the concurrency, the native driver is left in the dust, and the non-blocking EM/MySQL and DBSlayer take a commanding lead: ~2 seconds to complete all 200 requests! I've documented a DBSlayer EventMachine server implementation in a previous post, so let's take a look at how to interface with EM/MySQL:

require 'rubygems'
require 'eventmachine'
require 'evma_httpserver'
require 'em-mysql/lib/em/mysql'

class Handler  < EventMachine::Connection
  include EventMachine::HttpServer
  attr_accessor :db

  def process_http_request
    resp = EventMachine::DelegatedHttpResponse.new( self )

    EventedMysql.select("select sleep(1)") { |res|
      resp.status = 200
      resp.content = res
      resp.send_response
    }
  end
end

EventMachine.epoll
EventMachine::run {
  SQL = EventedMysql
  @mysql = EventedMysql.settings.update :host => 'localhost', :port => 3306, :database => 'dbslayer', :connections => 200

  EventMachine::start_server("0.0.0.0", 8083, Handler) {|conn| conn.db = @mysql}
  puts "Listening..."
}

There is still plenty of work that must be done to make EM/MySQL production ready, but it is definitely a promising project, and one to keep an eye on. Perhaps one day, we'll even have a drop in ActiveRecord adapter for asynchronous processing in Ruby 1.8 - wouldn't that be nice! Albeit, that's exactly where Neverblock is heading, if you're willing to make the switch to Ruby 1.9.

Ilya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.