Buildbot and Amazon Web Services Elastic Compute Cloud ("AWS EC2")

If you might care about Buildbot, you probably already know about it: "a system to automate the compile/test cycle required by most software projects to validate code changes," as the site puts it. It's written in Python with the Twisted framework.

At Canonical, I've been doing some work with it, and, lately, on it. My first accepted git branch added some bzr helpers. That was cool, but small. The one that was accepted this week is a bit more interesting, I think.

I've made it possible to hook up one or more on-demand, or "latent" slaves to the buildbot master. In the current single concrete implementation of the abstract class, for example, a latent slave always claims to be connected and ready to perform builds, even though it does not really have a remote connection to a machine ready to do work. But when a latent slave gets a build request, it instantiates an Amazon Web Services ("AWS") Elastic Compute Cloud ("EC2") virtual machine, using the image (and therefore operating system) of your choice to run the tests. It uses the nice AWS Python library, boto, to make the magic happen.

Implementing the same thing for other similar cloud computing services should be pretty easy. The buildslave.py module now has an AbstractLatentBuildSlave class. All you have to do to is subclass that and implement one method to start a virtual machine, and one to stop it. It's described in the pertinent section of the documentation.

I figure the new feature is probably only interesting for a relatively small subset of buildbot users. But it's still pretty cool, and a significantly different variation on what was there before. This is supposed to be released in 0.7.10, which is slated to be RSN, as I understand it.

If you are curious enough about it to want to poke around, the git master branch is here. You can find the EC2 latent slave in buildbot/ec2buildslave.py. The documentation also has extensive additions to try to help you get started.

Looks like I'll be doing a bit more buildbot work in February. Cool.