HTTP PubSub: Webhooks & PubSubHubbub

By on June 29, 2009

With all the recent buzz about real-time web, surely this is the year XMPP/AMQP Publish-Subscribe (PubSub) makes it to the big leagues! Or maybe not. Ejabberd (XMPP), RabbitMQ (AMQP) and other pubsub server implementations have come a long way but they remain cumbersome to setup and maintain, and perhaps more importantly, the clients require special libraries and a steep learning curve. That is not to say that either XMPP or AMQP are doomed for failure, in fact, they will continue to thrive, but there is a great case for a simplified PubSub implementation to cover the ad-hoc cases where a dedicated TCP channel might be an overkill: enter Webhooks.

The best part about Webhooks is that most of us are already familiar with them: callbacks over HTTP. Pioneered by PayPal and Subversion as a way to send real-time notifications to the client, they have found their way into many dozens of products we all use every day. Need pre or post commit hooks for your SVN or Git repository? Both GitHub and SVN support HTTP callbacks. Need a payment alert from PayPal, or an alert when a wiki page is modified? There are webhooks for that too. This simple mechanism allows us to build web services that work together via a simple and ubiquitous protocol we can all understand: HTTP!

Working with Webhooks

A good way to think about webhooks is as Unix pipes for the web. While PubSub is a great application of the technology, Webhooks allow bidirectional communication, which opens up our possibilities to: publishing notifications, chaining web services to perform complex actions, or allowing external plugins to enter the workflow. All of this functionality is accomplished via simple HTTP queries (both POST and GET), which means no special libraries or servers to make it all happen. In fact, building your own Webhooks powered PubSub server takes less than a hundred lines with the help of Sinatra, as demonstrated by the Julio Capote and his watercoolr project on GitHub. To see it in action, we can use PostBin as our mock recipient:

$ ruby postbin.rb http://www.postbin.org/1987c4m
# creating channel...
# adding subscriber to channel sej7u7
# {"status":"OK"}
# Post message: Hello Webhooks PubSub!
# {"status":"OK"}

In this scenario, our local watercoolr service acts as a PubSub server, accepting requests to create custom channels, maintaining a subscriber list, and pushing out notifications when an update arrives at the channel. Best of all, interacting with the server is as simple as writing a RESTful client:

require "rubygems"
require "rest_client"
require "json"

puts "creating channel..."
resp = RestClient.post "http://localhost:4567/channels", :data => ""
id = JSON.parse(resp)["id"]

puts "adding subscriber to channel #{id}"
resp = RestClient.post "http://localhost:4567/subscribers", {
  :data => {:channel => id, :url => ARGV[0]}.to_json
}

loop do
  print "Post message: "
  msg = STDIN.gets.chomp

  resp = RestClient.post "http://localhost:4567/messages", {
    :data => {:channel => id, :message => msg}.to_json
  }

  puts resp
end

PubSubHubbub Spec: Auth, Discovery & Protocol Implementation

Brad Fitzpatrick of Livejournal fame and Brett Slatkin, both of whom are currently at Google drafted a PubSubHubbub spec for a "simple web-scale pubsub protocol" which preserves the spirit of HTTP webhooks but adds a much needed layer of basic security, discovery, and coherent protocol definition:

We offer this spec in hopes that it fills a need or at least advances the state of the discussion in the pubsub space. Polling sucks. We think a decentralized pubsub layer is a fundamental, missing layer in the Internet architecture today and its existence, more than just enabling the obvious lower latency feed readers, would enable many cool applications, most of which we can't even imagine. But we're looking forward to decentralized social networking.

The google code project also provides an AppEngine implementation with a public hub, which we can test via an asynchronous (EventMachine based) PubSubHubbub Ruby library:

require "rubygems"
require "pubsubhubbub"

EventMachine.run {
  # publish single URL
  pub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish').publish "http://www.test.com/"
  pub.callback { puts "Successfully notified hub." }
  pub.errback  { puts "Uh oh, something broke: #{pub.response}" }

  # publish multiple URL's to hub
  feeds = ["http://www.test.com", "http://www.test.com/2"]
  pub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish').publish feeds

  pub.callback { puts "Successfully notified hub." }
  pub.errback  { puts "Uh oh, something broke: #{pub.response}" }
}
PubSubHubbub.git - Asynchronous PubSubHubbub Ruby Client

Webhooks & PubSubHubbub

Of course, neither Webhooks nor PubSubHubbub are the answer to every problem. Both XMPP and AMQP will continue to exist alongside, but chances are, will take on the brunt of high-velocity feeds while Webhooks can happily power the remaining 90% of the simpler use-cases. In fact, there is already a RabbitMQ PubSubHubbub hub server implemenation in the works! For further reading, flip through Jeff Lindsay's presentations on webhooks, and if you are interested in PubSubHubbub check out this presentation by Brad Fitzpatrick and Brett Slatkin.

Ilya GrigorikIlya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.