Aug 15

InterPlanetary File System (IPFS) and the (potential) future of the internet

I read an article called The Web We Have to Save recently. I really enjoyed it and it provided a lot of food for thought about the current state and future of the internet. I’ve already posted that link on Google+, but it seemed like as a good a time as any to start taking control of my own data on the internet again in the spirit of IndieWebCamp.

Somehow, I got to reading about IPFS (InterPlanetary File System), a new peer-to-peer protocol that looks like a really cool distributed file system. Decentralisation just seems like such a good idea – reduced latency, less centralised control and more much, much more permanence. Interesting and exciting stuff.

Nov 14

Tear-free Nvidia 770 on Linux

I finally gave up on the nouveau driver for now and have moved to the binary Nvidia one – version 343.22 at the time of writing.

This is mostly for my reference, but in case it’s useful to anyone else, here is a /etc/Xorg.conf.d/20-nvidia.conf that sets my graphics card to full performance when X starts. This gives me tear-free video using two (identical) monitors:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    Option         "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1"

I have an Nvidia GTX 770 and am running Arch Linux without a compositor.

Apr 13

Enabling keyboard mouse emulation out with Gnome/KDE

Gnome calls it “Mouse Keys” and it allows you to use something like xmodmap -e 'keycode 135 = Pointer_Button2' to make a key operate as a middle mouse button. If you don’t run Gnome/KDE though, you need to first turn on keyboard mouse emulation yourself by running:

xkbset m
xkbset exp =m

Goodbye gnome-settings-daemon, I won’t miss you.

Apr 13

User suspend/resume service scripts for systemd

I’m using Arch Linux these days, still with xmonad, and am trying to get away from as much Gnome stuff as possible. I have written these suspend and resume services for systemd to allow unprivileged users to suspend and lock the screen (using slock) and ensure that xmodmap runs on resume. They were adapted from the very useful Arch systemd wiki page and might come in handy for you too.


Description=User suspend actions




Description=User resume actions

ExecStart=/usr/bin/sleep 1 ; /usr/bin/xmodmap /home/%u/.Xmodmap


I’m not sure why I need to add the /usr/bin/sleep 1 in the resume service.

Don’t forget to enable them with systemctl enable suspend@yourusername.service and systemctl enable resume@yourusername.service as well as run systemctl --system daemon-reload whenever you change those files and something stuff like systemctl status resume@yourusername.service to help you debug them.

To suspend, run systemctl suspend as your unprivileged user. Zzz….

Jun 12

Booting F17 without an initramfs

I prefer not to use an initramfs with my laptop these days – it helps get my boot time down to just over 3 seconds (zooooom – check out these optimisations for more information). Here’s how I configured grub so that I don’t have to edit my grub config every time there is a kernel update. I made sure /etc/grub2/default has these lines:

GRUB_CMDLINE_LINUX="rootfstype=ext4 quiet libahci.ignore_sss=1 raid=noautodetect rd.lvm=0 SYSFONT=True KEYTABLE=uk rd.luks=0 LANG=en_US.UTF-8 rhgb quiet"

This isn’t yet quite enough, since I need to point grub at the correct root device. As things stand, grub2-mkconfig doesn’t seem to know about /dev/root.

Also, because kernel updates automatically provide an initramfs and /etc/grub.d/10_linux checks for the presence of that when generating config, I need to run a small script after each kernel update:

Here is /root/bin/move-initramfs-from-boot:

ln -s /dev/sda3 /dev/root && mv /boot/initramfs* ~/initramfs/ && grub2-mkconfig -o /boot/grub2/grub.cfg

UPDATE: my friend Graham has pointed out a nicer way to run scripts after install. Copy his method, I have!

May 12

$HOME/.ssh/ directory permissions

Because I always forget what permissions SSH needs to make public key authentication work:

chmod go-w $HOME/
chmod 700 $HOME/.ssh/
chmod 600 $HOME/.ssh/*

May 12

Our Git development workflow

Over the last couple of years we’ve been tweaking our development workflow and it has been really successful for us. It certainly isn’t revolutionary (it is heavily based on the Linux kernel), but it has worked so well and our code quality has improved so considerably that I feel it’s worth writing about. We are a very small team – typically 2 or 3 developers per-project – but I imagine this workflow will only get better at scale. A happy side-effect is that all of our projects are completely setup for remote working. I’m really loving this workflow.

Each project has a maintainer, who is also a developer. This role is simply to be the gatekeeper of origin/master of the Git repo. The maintainer is responsible for checking that a patch has the appropriate tags and is ready for inclusion (see the requirements in point 6 below). These concrete rules (which we do break, occasionally) force us to code review. Contrary to my expectations, I really enjoy the constant review process, from both sides.

We create a new local branch for every feature or bugfix. The workflow of a project is focussed around a PROJECT-devel mailling list. This is where all the patches are posted, reviewed and commented on before being pushed. The reviews are without doubt the biggest win here. We have adopted a strict “no change too small” policy – even a misplaced comma in a comment should be picked up on. This can feel pretty harsh at first, but I’ve found that we soon stop being precious and end up writing much better code anyway. Also, because we’re absolutely sure our commit messages will be read before even touching the repository, they have become a lot more detailed and actually useful.

For the nitty-gritty details, here’s the commit documentation from our projects in full:


Commit workflow

This project is run in a similar vein to the Linux kernel. You will need to
subscribe to <>.

The project is currently maintained by Mark Somerville <>.

The origin/master branch should _always_ be in a deployable state (and almost
certainly be what is actually deployed).

Some notes

* Most code changes should come with appropriate accompanying tests. This
  obviously doesn't apply to things like typo's and there are some other things
  where testing doesn't make sense.


Patches should come appropriately tagged. There are various tags that _should_
be used and others which aren't always required/appropriate.

* Signed-off-by: is the main one. There should be at least one of these on all
  patches by at least the author. Anyone else who handled or was involved in
  the patch or was in its delivery path should also add their Signed-off-by:
  e.g the committer.

  Signing-off a patch is an explicit acknowledgement that you know where it
  came from and that it is, to the best of your knowledge, eligible for
  inclusion in the project.

  The tag would look like:

  Signed-off-by: Mark Somerville <>

* Reviewed-by: says that the reviewer has performed a code review and that the
  changes make sense, look sane and are generally appropriate for inclusion.

  The tag would look like:

  Reviewed-by: Mark Somerville <>

* Tested-by: says that the tester has applied the patch and at least run all of
  the tests using whatever testing makes sense for the given context - it could
  be running `rake` or looking at it in a browser. More thorough tests can be
  performed, of course (and probably should be in some cases). The tag should
  relevant software and versions. For example, it will often include the Ruby
  interpreter and version used.

  We want to support at least MRI (Matz's Ruby Interpreter, the "standard"
  Ruby), version 1.8.7. Some interpreters are below:

  * MRI 1.9.3
  * MRI 1.8.7
  * RBX 1.8.7
  * JRuby 1.9.2

  The tag would look something like:

  Tested-by: Mark Somerville <> [MRI 1.9.3, RBX 1.8.7]
  Tested-by: Mark Somerville <> [RHEL5, MRI 1.9.3, Rails 3.0.11]

* Co-authored-by: is applied to patches that were written by more than one
  author. One example is pair-programming, where the code was created on one
  workstation. In this case the Git Author metadata is set to the owner of the
  workstation and this author would add a Signed-of-by tag, as usual. The other
  developer would add both a Co-authored-by and a Signed-off-by tag.

  The tag would look like:

  Co-authored-by: Mark Somerville <>

Tag ordering

Developers can and should add multiple tags to patches. The basic rule is that
a single developer's Signed-off-by tag should come after his others. If other
developers add tags to the patch, they should be added below.

Example: Mark was the author and Euan was the maintainer at the time:

  Tested-by: Mark Somerville <> [MRI 1.9.3]
  Signed-off-by: Mark Somerville <>
  Tested-by: Mike Lauder <> [MRI 1.8.7]
  Reviewed-by: Julia Allison <>
  Signed-off-by: Euan Maxwell <>

Example: Mark and Julia co-authored the patch on Mark's workstation and Euan was
the maintainer at the time:

  Tested-by: Mark Somerville <> [MRI 1.9.3]
  Signed-off-by: Mark Somerville <>
  Co-authored-by: Julia Allison <>
  Signed-off-by: Julia Allison <>
  Tested-by: Mike Lauder <> [MRI 1.8.7]
  Signed-off-by: Euan Maxwell <>

An exception to the Signed-off-by tag being at the end of a developer's tags
but above some others is when the author is also the project maintainer. In
this case, the author's Signed-off-by is the final tag, because he was the last
person to touch the patch (by committing it).

Example: Mark authored the patch and was also the maintainer at the time:

  Tested-by: Mark Somerville <> [MRI 1.9.3]
  Reviewed-by: Julia Allison <>
  Tested-by: Mike Lauder <> [MRI 1.8.7]
  Signed-off-by: Mark Somerville <>

Tag comments

On occasion, a comment should be added to a tag. Probably the most common use
of this is when the maintainer is ready to push the patch, but wants to perform
some minor tidy-up first (whitespace fixes, commit message alterations).
Because the change is minor this is easier than re-posting to the list.

The comments should be above the tag that is commented on and be of the form:

  [ just a wee comment, probably no more than a line ]

An example:

  Tested-by: Julia Allison <> [MRI 1.9.3]
  Signed-off-by: Julia Allison <>
  Tested-by: Mark Somerville <> [MRI 1.9.3]
  [ whitespace and refactored to idomatic Rails ]
  Signed-off-by: Mark Somerville <>

Merging a change

1. When you have made a change that you are happy with, you should commit it to
   your local Git repo with a Signed-off-by tag (see notes above). For commits
   involving code, you probably want to attach a Tested-by tag too.

   This is a good note about commit messages:

2. Generate a patch, probably using `git-format-patch` (or `git-send-email`

3. Email the patch to the PROJECT-devel mailing list (git-send-email is good
   for this) with the subject: "[PATCH] First line of commit"
   (`git-format-patch` will do this for you).

4. When your feature/fix spans multiple commits, you should probably email a
   patchset. This consists of a summary email with a subject like:

     "[PATCH 0/N] Add a teleporting time machine"

   with the commit patches as separate emails in reply to the summary.
   `git-send-email --compose` is very useful for this.

5. Developers comment on the patch(set). Sometimes this will be as simple as
   adding some tags to the commit. Multiple tags by different developers are
   fine (and welcomed).

   Other times, some changes will be needed before the team agree the patch is
   acceptable. If this is the case, you should incorporate the changes and go
   back to stage 2. The difference this time around is that when posting the
   patch during stage 3, the start of the subject should be changed to
   [PATCH v2] to indicate that this is the new version of the patch to review
   and test.

   You should also update the patch changelog. This versioning belongs below
   the commit message and above the diff stat (just below the ---). An example:

     Wrote pseudo-code algorithm for predicting lottery numbers

     Signed-off-by: Mark Somerville <>
     Co-authored-by: Euan Maxwell <>
     Signed-off-by: Euan Maxwell <>
      v2 - Added better comments clarifying the algorithm

      doc/lottery_number_prediction | 154 +++++++++++++++++++++++---------------
      1 files changed, 125 insertions(+), 29 deletions(-)

   When sending this v2 patch, be sure to email the patch as a reply to the
   appropriate patch thread, using the Message-ID you want to reply to.

6. To be acceptable for pushing to master, a commit must have at least one
   Signed-off-by tag (the author) and one or more Reviewed-by/Co-authored-by
   tags. There must also be at least one Tested-by tag that is by someone other
   than the original author (this is to protect against things only working due
   to a quirk of a single development or testing machine). Tested-by tags can
   be omitted for commits that don't need any testing or where it doesn't make
   sense, like documentation changes.

7. Once the patch is acceptable, it is the maintainers responsibility to apply,
   amend the commit to add the extra tags and push it to origin.

Mar 12

Finally fixing bufferbloat

I got a new server for the flat to replace my old, and long dead, Mini-ITX machine. It’s setup for various networking duties, the most important of which is the network routing. Now that I’ve got a real machine doing the routing I’ve been able to do some traffic shaping to mitigate the bufferbloat problem in the flat – something I’ve been wanting to do for a long time now.

While the general routing stuff was pretty straightforward, the traffic shaping stuff in Linux scared me. Fortunately, Wonder Shaper exists and saves me trying to wrap my head around tc. It really is rather nice – simply alter a few variables and latency stays not-shit, even when the connection is under heavy load. Ah… :)

My ping times to some Google server on an empty connection are about 18ms. Once another machine saturates the uplink, by uploading a large file, that rises to around 320ms! With traffic shaping, they only rise by a few ms.

EDIT: Take note of Dave Täht’s comments below (he is a lot more knowledgeable than me on these matters), highlighting problems with Wonder Shaper.

For my own benefit as much as anything else, here is my “just got it working” routing script:

# /etc/rc.d/rc.nat:

iptables -F
iptables -F -t nat
iptables -P INPUT ACCEPT

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

Now for the traffic shaping:

# /etc/rc.d/rc.bufferbloat
# Wonder Shaper
# please read the README before filling out these values 
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits. Also set the device that is to be shaped.

# Modem reports:
# DOWNLINK=21494
# UPLINK=2463



# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks

# low priority destination netmasks

# low priority source ports

# low priority destination ports

# Now remove the following two lines :-)

#echo Please read the documentation in 'README' first

if [ "$1" = "status" ]
	tc -s qdisc ls dev $DEV
	tc -s class ls dev $DEV

# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

if [ "$1" = "stop" ] 

###### uplink

# install root HTB, point default traffic to 1:20:

tc qdisc add dev $DEV root handle 1: htb default 20

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k

# From server to internel network

tc class add dev $DEV parent 1:1 classid 1:5 htb rate 100000kbit \
   burst 6k prio 1

# high prio class 1:10:

tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
   burst 6k prio 1

# bulk & default class 1:20 - gets slightly less traffic, 
# and a lower priority:

tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
   burst 6k prio 2

tc class add dev $DEV parent 1:1 classid 1:30 htb rate $[8*$UPLINK/10]kbit \
   burst 6k prio 2

# all get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10

# From server to internel network

tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
      match ip dst $IIP flowid 1:5

# TOS Minimum Delay (ssh, NOT scp) in 1:10:

tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
      match ip tos 0x10 0xfc flowid 1:10

# ICMP (ip protocol 1) in the interactive class 1:10 so we 
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
        match ip protocol 1 0xff flowid 1:10

# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:

tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8 0x10 0xff at 33 \
   flowid 1:10

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

# some traffic however suffers a worse fate
	tc filter add dev $DEV parent 1: protocol ip prio 14 u32 \
	   match ip dport $a 0xffff flowid 1:30

 	tc filter add dev $DEV parent 1: protocol ip prio 15 u32 \
	   match ip sport $a 0xffff flowid 1:30

 	tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
	   match ip src $a flowid 1:30

 	tc filter add dev $DEV parent 1: protocol ip prio 17 u32 \
	   match ip dst $a flowid 1:30

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

tc filter add dev $DEV parent 1: protocol ip prio 18 u32 \
   match ip dst flowid 1:20

########## downlink #############
# slow downloads down to somewhat less than the real speed  to prevent 
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
# attach ingress policer:

tc qdisc add dev $DEV handle ffff: ingress

# filter everything _from_ the Internet, drop everything that's
# coming in too fast:

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip dst \
   $EIP police rate ${DOWNLINK}kbit burst 10k drop flowid :1

# This reduces the queues in the driver buffer:
/sbin/ethtool -G eth0 tx 20
/sbin/ip link set dev eth0 qlen 4

Dec 11

Compiling 32bit Ruby on a 64bit machine

We achieved a reasonable improvement in out test suite run time (~10%) by running a 32bit Ruby 1.8.7 rather than 64bit. I’m not sure why. These instructions are for F16.

You need some 32bit libraries:

yum install glibc-devel.i686 openssl-devel.i686 zlib-devel.i686

Unfortunately, Ruby 1.8.7-p352 doesn’t seem to enjoy to being cross-compiled (at least on my setup). The patch for bug #5108 fixes this problem for me.

Next, configure the build. The most basic configure command that you need is probably something like this:

CFLAGS="-m32 -O2" LDFLAGS="-m32" CXXFLAGS="-m32" ./configure --prefix=/usr/local/ruby-1.8.7 --target=i686-unknown-linux-gnu

After building and installing, you will probably want to install RubyGems again and will need to rebuild any C extensions you use. Of course, building them will require any associated 32bit libraries. The mysql gem is slightly trickier than some:

yum install mysql-devel.i686
gem install mysql -- --with-mysql-config=/usr/lib/mysql/mysql_config

Jan 11

Deleting tracks from within Rhythmbox

I was having trouble removing tracks in Rhythmbox (right-click “Move to trash”). Running the application with debugging enabled didn’t do too much help, but the many bug reports of similar problems did.

I have my music stored on a separate EXT4 partition rather than the commonly reported NTFS, so the fix was as simple as creating a .trash-{uid} directory in the root of the mounted partition with the correct ownership and permissions, of course.

Dec 10

Urchin 0.1.0 released

A few months ago I wanted to teach myself some new shell tricks. One thing led to another and I ended up writing my own interactive shell in Ruby. Urchin exists mostly just for the sake of it, but I’m looking forward to experimenting and implementing some things that aren’t found in other shells.

Urchin does not aim to be POSIX compliant. Finding POSIX shell scripting pretty nasty was one reason I started this project. However, I generally really like the existing standard shells, so many things work in a way you would expect: pipelines, redirections, job control, globbing, environment variables, aliases and tilde expansion are all there. Documentation, not so much!

Urchin recently passed my criteria for a first release – “run as my login shell for a week without crashing or annoying me”. For sure, there are some problems (tab-completion, quoting and escaping issues spring to mind), but I had to draw a line somewhere and just release something.

Trying it out

This still feels very much like a beta release. The chances are you don’t use a shell exactly like me and this isn’t yet a viable Bash/Zsh replacement.

I’ve only tested this release on Linux with GNU Readline. I’d love to hear how it runs on other platforms. Future versions will be tested on more platforms.

Warning: I’d advise testing it out for a while before you get too trigger happy and chsh!

If I haven’t managed to put you off yet, follow the instructions on the Urchin homepage to try it out.

Nov 10

Adding a third mouse button to an Acer Aspire One

To set my strange menu-looking key on my Acer Aspire One to be a middle mouse button, I use this:

xmodmap -e "keycode 135 = Pointer_Button2"

It is important to enable mouse keys for this to work (that’s the bit I always forget).

May 10

Making Chromium use correct default applications

Loads of people seem to be having this problem, but not many people actually seem get around it. This works for me.

My environment:

  • Fedora 12 x86_64
  • Chromium 5.0.382.0
  • Running xmonad window manager

Chromium uses xdg-open to open file types it isn’t associated with. After a quick dig through the script, I saw that xdg-open tries to open the file using mimeopen, falling back to using run-mailcap and finally resorting to calling `$BROWSER $1`.

I don’t have mimeopen or run-mailcap installed and I noticed that my $BROWSER environment variable wasn’t set. When that is the case xdg-open looks up and tries to use a basic list of web browsers, one of which is htmlview (this is why Chromium is launched when you try to view PDFs).

I simply set my $BROWSER environment variable to gnome-open, which is smart enough to know what to with a file or URI. If you go down this route, remember to make sure the variable is set for all your browser shells.

Apr 10

Unix signal programming in Ruby

I recently thought I would be able to use Unix signals to solve a problem in a Ruby program I’m writing. It turned out not to be workable, but was a fun journey into Unix signal handling and how they work (or don’t) with MRI.

I’m not a signals expert – corrections and opinions are very welcome! Also, I started to feel a bit out of my depth when patching the longjmp() calls. I’d love to find out more about this stuff.

SignalPhoto by David Blaikie

Brief signals primer

Signals are used to alert processes or threads about a particular event. Synchronous signals are usually the result of errors in executing some instruction (such as an illegal address reference) and are delivered to the thread that caused the error. Asynchronous signals are external to the execution context and are probably the ones you’re more familiar with – they can be sent between processes using things like kill or delivered when needed by the kernel.

When a signal is generated it is immediately put into the “pending” state. If the process has a thread that has not blocked signals of that type, it is delivered straight away. If that type of signal is blocked by all threads in the process, it remains pending until they are unblocked in one of the threads, at which point it is delivered immediately. Delivered signals can be ignored (often the default response) or processed by a signal handler. In Ruby, we define a SIGUSR1 handler like this:

Signal.trap("USR1") do
  puts "USR1 caught"

Why might we want to block signals?

Blocking signals is often used when we have a section of code that must not be interrupted. To enable this, each thread maintains a signal mask. This is the list of signal types that the thread is blocking, which we can examine and change using pthread_sigmask() (sigprocmask() in single-threaded programs). A new thread inherits the signal mask from the parent. However, each thread does not have its own set of signal handlers – these are shared throughout the process. Asynchronous signals that are delivered to the process can be processed by any thread that has not blocked those signals.

Signals in MRI

Unfortunately, MRI isn’t really very friendly to Unix programmers wanting to play with the signal mask, as we’ll see.

MRI defines the Signal module, that only contains two methods: Signal.trap and Signal.list, which provides the mapping of signal names to numbers for your platform. Since none of the other libc signal handling functions are defined, I created a library to provide them (and some other system calls sometime). syscalls is built using the lovely FFI library. This mirrors the libc functions closely, with a couple of Ruby-style shortcuts added.

Ruby 1.8

As Joe Damato found, MRI 1.8 with pthreads enabled is rather rt_sigprocmask() happy. It seemed obvious that all that mucking about with the signal mask would cause strange behaviour when blocking signal, but let’s see how:

require "syscalls/signal"

mask =
Syscalls.sigaddset(mask, "USR1")

puts "Block and roll!"
Syscalls.sigprocmask(Syscalls::SIG_SETMASK, mask, nil)

puts "Looks fine so far - let's raise an exception..."


puts "Aw-naw!"

What’s going on?

Using strace with ruby 1.8.6 (2009-08-04 patchlevel 383) [x86_64-linux] gives us:

write(1, "Block and roll!\n", 16Block and roll!
)       = 16
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Looks fine so far - let's raise "..., 48Looks fine so far - let's raise an exception...
) = 48
rt_sigprocmask(SIG_BLOCK, NULL, [USR1], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Aw-naw!\n", 8Aw-naw!
)                = 8

Line 3 is from Ruby calling getcontext – it is pretty harmless since it is passing SIG_BLOCK along with an empty set of signals, which adds nothing to the existing signal mask. Line 4 is our own call to sigprocmask – note we’re using SIG_SETMASK, which replaces the existing mask. So far, so good. However, on lines 7-9 Ruby stores the old mask (our SIGUSR1), replaces it with an empty mask and then immediately replaces that with our SIGUSR1 mask again.

But, the mask is only empty for a fraction of a second – I think I’ll be alright!

Think again! It’s tempting to think that this wouldn’t be a problem in most real-world situations, but you may recall that when a signal cannot be delivered because it’s blocked it is put into a pending state. When that signal type is unblocked, the signal is immediately delivered. This means there can actually be plenty of time to queue up a signal to cause a problem here.

The REE stuff below all applies to the other MRI 1.8 flavours I tested too – that’s 1.8.{6,7} on 64-bit Linux.

REE 1.8.7-2010.01

REE has Joe’s --disable-ucontext patch applied, which meant a lot fewer sigprocmask()s to wade through! In fact, it nearly worked – just our old SIG_SETMASK friend set during the exception handling:

write(1, "Block and roll!\n", 16Block and roll!
)       = 16
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Looks fine so far - let's raise "..., 48Looks fine so far - let's raise an exception...
) = 48
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "Aw-naw!\n", 8Aw-naw!
)                = 8

Time to dig around and see why Ruby is doing that.

Calling raise resulted in a call to rb_longjmp(), which appears to be a reimplementation of siglongjmp() (or _longjmp() – I don’t know which). This in turn calls rb_trap_restore_mask(), which sets the signal mask back to the mask that was stored when Ruby starts up or the last call to Signal.trap was made.

Patching Ruby 1.8 – including 1.8.6, 1.8.7 and REE

This might be dangerous or not even sensible. Let me know if you find out!

As far as I can tell, simply removing the call to rb_trap_restore_mask() shouldn’t break anything since the places that the trap_last_mask variable is set are very limited. It may not the best place for the fix (if rb_longjmp() is actually siglongjmp(), this might break the reimplementation), but it does at least appear to work.

Here’s the truly tiny patch.

Ruby 1.9

Ruby 1.9.1-p376 is a slightly more tricky case. As you’ll be aware, MRI 1.9 maps each Ruby thread to a native C thread and uses the GIL to ensure only one runs at any one time. The interpreter uses a thread to trigger an interrupt in order to schedule threads. This thread is created on initialisation of the interpreter and means that even very simple programs have two native threads running.

As we can see in the abbreviated strace below, the signal mask is empty when the timer thread is created and this will be inherited. This means that if we block a signal type in our main Ruby thread, they will still be able to be delivered and handled by the timer thread.

Below that we can see rb_trap_restore_mask() emptying the mask when it is called from rb_longjmp(). The sigaltstack() and following sigaction()call on lines 4-5 tell Ruby to handle segfaults on a different stack.

rt_sigaction(SIGHUP, {0x48b9f0, [], SA_RESTORER|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR1, {0x48b9f0, [], SA_RESTORER|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8) = 0
sigaltstack({ss_sp=0x1b0baf0, ss_flags=0, ss_size=16384}, {ss_sp=0, ss_flags=SS_DISABLE, ss_size=0}) = 0
rt_sigaction(SIGSEGV, {0x48bce0, [], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clone(Process 23020 attached
child_stack=0x7f3b2c188ff0, flags=CLONE_VM|CLONE_FS| CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM| CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f3b2c1899e0, tls=0x7f3b2c189710, child_tidptr=0x7f3b2c1899e0) = 23020
[pid 23019] write(1, "Block and roll!", 15Block and roll!) = 15
[pid 23019] write(1, "\n", 1
)           = 1
[pid 23019] rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
[pid 23019] write(1, "Looks fine so far - let's raise "..., 47Looks fine so far - let's raise an exception...) = 47
[pid 23019] write(1, "\n", 1
)           = 1
[pid 23019] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 23019] write(1, "Aw-naw!", 7Aw-naw!)      = 7
[pid 23019] write(1, "\n", 1
)           = 1

Patching Ruby 1.9.1

First off, we need to mask all signals as soon as the timer thread is created. This makes sure that all signals can only be delivered to the main Ruby thread (until we create some more). I suppose this will actually slow down signal delivery by some small amount.

Ruby itself blocks some signals for a time to allow sections of code to run without interrupts, using rb_disable_interrupt() and rb_enable_interrupt(). These functions mask and unmask all signals. We need to make Ruby save the existing signal mask while it blocks all signals, then restore the old mask, rather than unblocking everything.

Here’s the patch. Again, I don’t think this has any negative side-effects, but it also might not be a good idea. Let me know if you find out!

Further reading

Mar 10

Ruby process management with Jobby

Jobby is a generic forking process manager written in Ruby, built with robustness as a primary goal from the start. It was initially developed for offloading long running background tasks from the webserver in Rails applications, but it turned out to be useful in its own right, so we extracted it to work more generally. It uses Unix sockets and a call to fork(), so won’t work on Windows or JRuby, but has been tested on several flavours of Linux and OS X. It is also copy-on-write friendly, should the Ruby interpretter that’s used support it.

If you prefer to read code than blog posts, server.rb does most of the heavy lifting.

After a long struggle of using the old BackgrounDRb, last year we finally threw in the towel. We checked out the alternatives, but nothing really fitted the bill. Since background job processing is quite central to some of our applications, we built our own. From the start we had some requirements:

  • Run jobs in parallel up to a maximum number or processes, then start queuing
  • Robust – it just had to be solid
  • Easy to manage – we hated those BackDRb scripts
  • Reliable logging
  • Able to get messages from the background processes

Jobby itself does not provide a mechanism for getting messages from the child processes. I’ll detail a jobby_rails plugin that handles that some other time.

Installing Jobby

gem install jobby will do the trick if for some reason you don’t hate gems. You can also check out the source from github.

Gentoo users can install Jobby from the Gentoo Ruby overlay. Note: I stopped using Gentoo sometime around March 2010. It might still be there.

How does it work?

Jobby is a command line program. It consists of a daemon that listens on a Unix socket and will fork when it receives a connection. The forked child either runs some Ruby code or executes some shell code, depending on how the daemon was started. When you call Jobby, you pass a single parameter to STDIN – this string is passed to the forked child. It could be an ID of some sort, a whole email, a marshalled object or whatever.

There is only one command used, jobby. When this is run it first checks for the existence of the daemon process listening on the socket. If it’s not there, it starts one (and then executes the child command). This is great because it makes daemon startup totally automatic and if the daemon should go down (this hasn’t happened to us yet in the months that we’ve been using it), the next request will start up a new one.

Stopping Jobby

There are two ways to stop the daemon, both using Unix signals. Issuing a USR1 initiates a “very friendly shutdown”, which will stop the daemon accepting any new connections but will allow it to finish the jobs in the queue before terminating. Issuing a TERM signal will stop the daemon forking, then issue a TERM signal to the child processes and terminate the daemon.

Other nifty stuff

When using Jobby to run Ruby code, you might want to have the children load a few libraries before they do any work. Repeating this every time is a pain, so you can instead pass a --prerun parameter to prerun some Ruby code in the daemon before any forking takes place. This will load the libraries (at least in this example) so that they are immediately available in the children when the fork occurs. If you’re using a copy-on-write friendly interpretter, like Ruby Enterprise Edition, you’ll get the memory saving benefits of that too.

You can run multiple Jobby daemons – just call each one by passing a different socket parameter. This is how you can run different code for different job types that you have.

To make your debugging life easier, the forked children will show up in process listings as jobby: /path/to/socket. The actual daemon process will show up as jobbyd: /path/to/socket.

Using Jobby with Rails

Jobby was designed for use in Rails applications, but was extracted since it is generally useful. This post only describes Jobby itself, which you can happilly use from a Rails app using a system call. Sometime I’ll get round to releasing and detailing a Rails plugin for Jobby that adds a communication layer to Jobby and makes Rails intergration nicer. We’re using it just now, but it’s not really releasable as it is. I decided that since the main Jobby code has been so stable for us, it made sense to release it now.

Nov 09

Ensuring epatch works correctly on embedded Gentoo

On my BeagleBoard, the ebuild messages were telling me that the patches had been applied successfully, but inspection of the code showed they hadn’t. It was a nasty problem to fix – it seems like a epatch wasn’t liking the cross-compiled bash.

After compiling bash natively and getting a another shell, the epatching was successful.

Dec 07

Mercurial 0.9.5 for Maemo

I’m using Mercurial to track changes in my latest project, a GTK mapping application. I couldn’t find a build for my N800, so I made one myself. I’ll host a .deb until I get around to putting it on the garage or creating a repository.

Edit: I’ve started a garage project.

To install python2.5-runtime I needed to add to the application catalogue to satisfy some dependencies. Tested with OS2008 beta on an N800 only.

Jun 07

Notes on setting up Gentoo

I’m installing Gentoo on my laptop – here are some notes (for myself, really).

  • Gentoo installs the flash plugin to a path with “netscape” in it. There is a problem with this so we need to symlink the plugin to the opera plugin path.

Aug 06

Remapping caps lock (again)

I was using xmodmap to re-map the caps lock. I’m not sure why – a better way, which means it also works on the console, is to alter the keymap file. For me (using Gentoo, UK keymap) this was as simple as:

  1. cd /usr/share/keymaps/i386/qwerty
  2. gunzip
  3. vim (change “Caps_Lock” to “Escape”)
  4. gzip
  5. loadkeys

May 06

Using Rake to automate PHP testing on linux

I’m getting really into Rake, a build tool like Make written in (and therefore able to use the full power of) Ruby. Martin Fowler has a nice article about it.

I’ve been doing some PHP work again recently and, inspired by autotest, knocked together a Rake task which monitors directories and runs my tests when it notices a change. Unlike autotest, which checks for changes every few seconds, this task uses inotify and is rapido enough that it always manages to start the tests before I can switch consoles. Of course, this can be used for running anything, not just the PHP tests I’ve been using recently.

Aside: Simpletest is the nice PHP testing package that I use.


  • Linux kernel with inotify (version 2.6.?+)
  • Ruby – only tested with 1.8.4
  • Rake – only tested with 0.7.1
  • ruby-inotify – only tested with 0.3.0, older versions will probably not work

 1 desc "Run the tests." 
 2 task :test do
 3   # assign "out" so that test errors don't cause an exit
 4   out = system "php test/init.php" 
 5 end
 7 desc "Run tests automatically when files change" 
 8 task :keep_testing => [ :test ] do
 9   require 'io/INotify'
10   require 'find'
12   # allows us to re-run the tests
13   class Rake::Task
14     attr_accessor :already_invoked
15   end
16   Rake::Task[:test].already_invoked = false
18   inotify =
20   thread = do
21     inotify.each_event do |event|
22       # .swp files are used by Vim
23       if event.type == 'modify' and event.filename !~ /.swp$/
24         puts "#{event.filename} modified..." 
25         puts "===" 
26         Rake::Task[:test].invoke
27         Rake::Task[:test].already_invoked = false
28         puts "" 
29       end
30     end
31   end
33   # default list of directories
34   ENV["directories"] ||= 'lib test'
36   # start watching the directories
37   ENV["directories"].split.each do |directory|
38     Find.find(directory) do |file|
39       if ['.svn', 'CVS', 'RCS'].include? File.basename(file) or ! file
40         Find.prune
41       else
42         begin
43           puts "Adding #{file}" 
44           inotify.watch_dir(file)
45         rescue
46           puts "Skipping #{file}: #{$!}" 
47         end
48       end
49     end
50   end
51   puts "" 
53   thread.join
54 end

The default list of directories (relative to the current directory) is specified on line 34. You can override this at runtime by specifying the “directories” environment variable.

Jan 06

Remapping caps lock in X

Really just posting this so that I can find it again when I need it. I spend most of my day in Vim and never use the caps lock. I finally got round to re-mapping it to the escape button by using xmodmap like so:

xmodmap -e 'keysym Caps_Lock = Escape' -e 'clear Lock'

Just got to re-map my fingers now.

Apr 05

Ruby, Rails and some KDE stuff

Like many, I’ve been checking out the Ruby on Rails framework. Jonathan at work had reckoned that it was worth investigating to see how good it is for quickly deploying content managed web apps. For the last couple of weeks I have been developing a project management thing for us to use internally in Rails and I’m starting to get a feel for how things are pieced together. It’s really quite nice and getting something quick and simple up is Formula 1 fast. Once things become more complicated the development slows to a more sedate Subaru Impreza speed. Still quick. I know that it’s something that could be done in other languages, as some blog posts have pointed out, but that is missing the point – it hasn’t been done before, at least not as nicely as this. I’m certainly no expert on develpment frameworks, but the array of built in functions and facilities is impressive, and growing. This is the nicest thing about Rails – it allows you to get on with programming the site, rather than buggering about setting stuff up and performing database queries. And what a joy programming it is, largely thanks to Ruby.

Ruby is a pleasure to use and, in many ways, I’ve taken to it more than Rails. After a couple of WTF? hours at the start, things started to make sense. Then I got hold of The Pickaxe and things really started making sense. The more I poked and prodded Ruby, the more I liked it. Small and concise blocks of code started doing some pretty smart stuff. I ported a wee PHP-GTK program that I have written over to Ruby and it was around 30% smaller – not bad for programming a new language, day 1 (this is a testament to Ruby, not my talent). Ruby is a beautiful object-oriented language – it’s the best thing since sliced linux.

Recently at work I have had to develop some sites remotely, via FTP – a nasty practice that I don’t enjoy. It hasn’t been made easier by the fact that I’ve been unable to find any decent FTP program for KDE/linux (a minor untruth – Kasablanca is nearly OK). So I have decided to write my own, using Ruby and Korundum. I’ll write more about this little project later, but I will say that it’s a complete pleasure to write an application after writing websites. No browsers, no CS-fucking-S, ahh….

Another KDE thing that is on the horizon (but, for some reason, hasn’t had as much coverage as I would have expected) is the forthcoming Mozilla/Gecko Kpart. This should give the ability to swap between using KHTML and gecko for rendering at the touch of a button in Konqueror (or any other Kpart aware browsers) and should be a great feature for web developers, or anyone who likes one browser, but prefers the rendering in another etc… Very cool.

Mar 05

I’m looking for a new host

There’s nothing particuarly wrong with my current host (in fact, they’re pretty good), but I’m after a few things that they just don’t provide. Unfortunately, I think find all of this stuff at a reasonable rate might not be that easy. Here’s the essential stuff:

  • Apache 2
  • PHP5
  • MySQL 4
  • mod_ruby
  • shell access
  • OpenSSL
  • IMAP mail
  • Sensible space and bandwidth, nothing crazy

Mar 05

Gentoo == XR3i?

I found this piss take of Gentoo today, I think it’s thing Alex was telling me about. Pretty funny, even if you can’t find time to read the lot. Perhaps it’s sad/worrying that I agree with a lot of the quotes. A couple of the best ones are, IMO:

“I essentially started using Gentoo because my ….ing KDE clock would never show the right time in Red Hat.”

“What other linux will let you have a vector optimized wordprocessor?”

Mar 05

KDE 3.4

I’ve been running Gentoo and KDE 3.4 on my new laptop for a week or two now and am super-impressed. Since I installed KDE 3.4 RC1 onto my new laptop, I assumed that the whole desktop was looking so nice just because of the wicked screens that Sony put in their computers, but having read a few reports from others that it generally looks way nicer than before shows that KDE has made big improvements too. There’s nothing really in particular that you can pick out, but the whole thing just feels right, if that makes any sense. I liked 3.3, but this is a major improvement for me. It’s fast too, although how much of that is down to my new processor if hard to say.

I tried Kmail in 3.3 but found it horrifically unstable when using IMAP. 3.4 also suffered from this when I used normal IMAP, although I’m happy report that the “disconnected IMAP” settings are at least fifty times more stable (at least for me). Kmail is the best e-mail program I’ve ever used – very cool. I’ve never used Kontact before, but it’s pretty nice. Last year Akregator was in beta when I used it and, to be honest, it’s not all that much better now. It’s got potential though, there’re just a few little things that are holding it back a bit. Konqueror is much faster on this machine for me (still doesn’t touch Opera for speed). I use it as my main browser now, mainly because of the way everything interacts so damn well. This is the clincher for a lot of KDE programs IMO and it makes the whole KDE thing so good. Kate, Kopete are both pretty damn cool programs too.

Plastik is in the default themes now, yippee. Although they removed the best feature of Plastik – being able to close a window with your mouse in the very top right of the screen – WHY?