March, 2010

Mar 10

Ruby process management with Jobby

Jobby is a generic forking process manager written in Ruby, built with robustness as a primary goal from the start. It was initially developed for offloading long running background tasks from the webserver in Rails applications, but it turned out to be useful in its own right, so we extracted it to work more generally. It uses Unix sockets and a call to fork(), so won’t work on Windows or JRuby, but has been tested on several flavours of Linux and OS X. It is also copy-on-write friendly, should the Ruby interpretter that’s used support it.

If you prefer to read code than blog posts, server.rb does most of the heavy lifting.

After a long struggle of using the old BackgrounDRb, last year we finally threw in the towel. We checked out the alternatives, but nothing really fitted the bill. Since background job processing is quite central to some of our applications, we built our own. From the start we had some requirements:

  • Run jobs in parallel up to a maximum number or processes, then start queuing
  • Robust – it just had to be solid
  • Easy to manage – we hated those BackDRb scripts
  • Reliable logging
  • Able to get messages from the background processes

Jobby itself does not provide a mechanism for getting messages from the child processes. I’ll detail a jobby_rails plugin that handles that some other time.

Installing Jobby

gem install jobby will do the trick if for some reason you don’t hate gems. You can also check out the source from github.

Gentoo users can install Jobby from the Gentoo Ruby overlay. Note: I stopped using Gentoo sometime around March 2010. It might still be there.

How does it work?

Jobby is a command line program. It consists of a daemon that listens on a Unix socket and will fork when it receives a connection. The forked child either runs some Ruby code or executes some shell code, depending on how the daemon was started. When you call Jobby, you pass a single parameter to STDIN – this string is passed to the forked child. It could be an ID of some sort, a whole email, a marshalled object or whatever.

There is only one command used, jobby. When this is run it first checks for the existence of the daemon process listening on the socket. If it’s not there, it starts one (and then executes the child command). This is great because it makes daemon startup totally automatic and if the daemon should go down (this hasn’t happened to us yet in the months that we’ve been using it), the next request will start up a new one.

Stopping Jobby

There are two ways to stop the daemon, both using Unix signals. Issuing a USR1 initiates a “very friendly shutdown”, which will stop the daemon accepting any new connections but will allow it to finish the jobs in the queue before terminating. Issuing a TERM signal will stop the daemon forking, then issue a TERM signal to the child processes and terminate the daemon.

Other nifty stuff

When using Jobby to run Ruby code, you might want to have the children load a few libraries before they do any work. Repeating this every time is a pain, so you can instead pass a --prerun parameter to prerun some Ruby code in the daemon before any forking takes place. This will load the libraries (at least in this example) so that they are immediately available in the children when the fork occurs. If you’re using a copy-on-write friendly interpretter, like Ruby Enterprise Edition, you’ll get the memory saving benefits of that too.

You can run multiple Jobby daemons – just call each one by passing a different socket parameter. This is how you can run different code for different job types that you have.

To make your debugging life easier, the forked children will show up in process listings as jobby: /path/to/socket. The actual daemon process will show up as jobbyd: /path/to/socket.

Using Jobby with Rails

Jobby was designed for use in Rails applications, but was extracted since it is generally useful. This post only describes Jobby itself, which you can happilly use from a Rails app using a system call. Sometime I’ll get round to releasing and detailing a Rails plugin for Jobby that adds a communication layer to Jobby and makes Rails intergration nicer. We’re using it just now, but it’s not really releasable as it is. I decided that since the main Jobby code has been so stable for us, it made sense to release it now.