Extending Tokyo Cabinet DB with Lua
Tokyo Cabinet is a trove of hidden of gems, the more you learn about it, the more you will appreciate the design and technical decisions behind it. By database standards it is a young project (started in 2007), but since it is a successor to the QDBM project developed by Hirabayashi-san (2000-2007), we could make the argument that it has been, in fact, nine years in the making.
Best of all, the rewrite allowed the project to shed its past baggage and build on a more modern stack with a better perspective of the required features for a modern deployment (Drizzle is another recent example from the MySQL camp). Case in point, Tokyo Tyrant supports Lua scripting on the server, allowing us to add arbitrary user defined functions (UDF's in MySQL parlance) and to extend the database itself!
Lowering the Barrier With Lua Scripting
If you have played with Lego Mindstorms, or ever tried to script your phone or a gaming environment (WoW), chances are you've worked with Lua. Due to its extremely small footprint (~212kb for source, documentation, and examples), fast interpreter, portability and easy to understand syntax it has become the de-facto embedded scripting language for many applications. Explore the available tutorials, or pick up "Programming Lua" by the creator of the language for a more in depth look, it's a fun language!
So why is Lua scripting such an exciting feature for Tokyo Tyrant? Because it lowers the barrier for programming and extending the database by an order of magnitude! If you have ever worked with MySQL UDF functions, you'll definitely appreciate the ease and the speed of development. No need for mucking with internal API's, nothing to compile or link against, and execution is done within a stable and sandboxed environment.
Extending Tokyo Tyrant with Lua
Anytime you bootup a Tokyo Tyrant server you can tell it to load arbitrary Lua code alongside which will then be evaluated at runtime if the client requests it. From there, the developer of the extension has full access to the incoming request and the underlying Tokyo Cabinet database, allowing us to inject arbitrary functionality. To get a flavor for the workflow, take a look at some of the examples in the following slides:
Extending Tokyo Cabinet with Lua: InfoQ video from FutureRuby '09 (~30 mins)
tokyo-recipes.git - Lua Extensions for Tokyo Tyrant
Because the Lua scripting interface is relatively new, the number of available extensions and documentation is not very large. For that reason, I've started the tokyo-recipes repo on GitHub, in which I've aggregated some of the available extensions, and added extra documentation and examples to help grease the wheels: TTL functionality (ala memcached), working with Sets (ala Redis), session timestamping and a wordcount map-reduce example just to name a few. Give it a try, it is an extremely powerful feature!
Ilya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.