Ruby Web-Services with Facebook's Thrift

By on November 30, 2007

Facebook's Thrift framework made a nice splash in the news on Wednesday with an official announcement of the move to the Apache Incubator. This is certainly exciting news for many developers as Thrift is arguably one of the best (recent) lightweight systems for cross-language programming using code-generation, RPC, and object serialization. Designed from the ground up by the Facebook development team it enables seamless integration of the most commonly used languages (Ruby, Perl, C/C++, Haskell, Python, Java, and more) via a set of common RPC libraries and interfaces. Not to mention, the guys at Facebook know a thing or two about high-performance websites, and Thrift does not disappoint!

Defining a service in Thrift

The core goal of Thrift is to enable efficient and reliable communications across multiple languages. All datatypes and services are defined in a single language-neutral file and the necessary code is auto-generated for the developer by the 'thrift compiler'. The beauty of this approach is, of course, the ability to mix and match implementations of services: your server may be written in C++, but you can access it seamlessly via a Python, Java, or a Ruby client. Thrift will take care of the communications links, object serialization, and socket management! Having said that, nothing stops you from using the same language on both the client and the web-server. Let's take a look at a Ruby-specific example.

Assuming you've downloaded, compiled, and installed the thrift libraries (./configure && make && make install), we can jump right into the definition of our new service:

# Define a struct, an exception and a 'quick service - QService'
# - To learn more about Thrift's datatypes, head to:
# - http://developers.facebook.com/thrift/

 struct Lookup {
  1:string bucket,
  2:string key
 }

 exception InvalidKey {
  1: string error
 }

service QService {

  /**
   * A method definition looks like C code. It has a return type, arguments,
   * and optionally a list of exceptions that it may throw. Note that argument
   * lists and exception lists are specified using the exact same syntax as
   * field lists in struct or exception definitions.
   */

   i32 exponent(1:i32 base, 2:i32 exp),
   string get_key(1:Lookup l) throws (1:InvalidKey e),
   async void run_task()

}

We've defined three different functions: remote exponentiation, a key lookup, and an asynchronous method call. To generate the required code, we simply call the thrift generator:

# Generate C++, Ruby, and Python implementations
#  - generated code will be in 'gen-cpp', 'gen-rb', 'gen-py'
$ thrift -cpp -rb -py qservice.thrift

Implementing a Thrift powered server in Ruby

Thrift does most of the work, and we just need to provide the actual implementation of our functions in either Python, Ruby or C++. A quick and dirty Ruby implementation might look something like this:

# include thrift-generated code
$:.push('../gen-rb')

require 'thrift/transport/tsocket'
require 'thrift/protocol/tbinaryprotocol'
require 'thrift/server/tserver'

require 'QService'

# provide an implementation of QService
class QServiceHandler
  def initialize()
    @log = {}
    @hash = {'bucket1' => {'key1' => 'value1'}}
  end

  def exponent(base, exp)
    print "#{base}**#{exp}\n"
    return base**exp
  end

  def get_key(lookup)
   if @hash[lookup.bucket][lookup.key]
     return @hash[lookup.bucket][lookup.key]
   else
     e = InvalidKey.new
     e.error = 'Cache miss'
     raise e
   end
  end

  def run_task()
    print "kick off long running task...\n"
  end

end

# Thrift provides mutiple communication endpoints
#  - Here we will expose our service via a TCP socket
#  - Web-service will run as a single thread, on port 9090

handler = QServiceHandler.new()
processor = QService::Processor.new(handler)
transport = TServerSocket.new(9090)
transportFactory = TBufferedTransportFactory.new()
server = TSimpleServer.new(processor, transport, transportFactory)

puts "Starting the QService server..."
server.serve()
puts "Done"

To expose our QService to the outer world, we wrap it into a TCP socket listening on port 9090. From this point on, Thrift takes over all communications, serialization and handling of the incoming requests. Because the protocol is identical in every language, the client may be written in any language of choice.

Building a Ruby client

In similar fashion, we can build a Ruby client for any Thrift service with just a few lines of code. To interface with our server implementation above, we can do the following:

$:.push('../gen-rb')

require 'thrift/transport/tsocket.rb'
require 'thrift/protocol/tbinaryprotocol.rb'
require 'QService'

begin
  transport = TBufferedTransport.new(TSocket.new('localhost', 9090))
  protocol = TBinaryProtocol.new(transport)
  client = QService::Client.new(protocol)

  transport.open()

  # Run a remote calculation
  answer = client.exponent(50,2)
  print "50**2=", answer, "\n"

  # Run a 'cache' lookup
  lookup = Lookup.new()
  lookup.bucket = 'bucket1'
  lookup.key = 'key1'

  print "Lookup: ", client.get_key(lookup), "\n"

  # Force a cache miss
  begin
    lookup.bucket = 'bucket2'
    print "Lookup: ", client.get_key(lookup), "\n"
  rescue InvalidKey => e
    print "InvalidKey: ", e.error, "\n"
  end

  print client.run_task()

  transport.close()

rescue TException => tx
  print 'TException: ', tx.message, "\n"
end

Every language has its strong points, and the ability to mix and match server/client implementations on the fly becomes a nice bonus - kudos to the Facebook developers for open-sourcing this gem. Thrift is certainly a project to keep a close eye on, as it may well become your swiss-army knife when it comes to distributed web-service development.

thrift-ruby.zip - Thrift definition + Ruby server/client

Additional resources and reading:

Ilya GrigorikIlya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.