Open Source

Canonical has encouraged us employees to blog about the work we do on open source. I intended to include these comments as part of the previous post, but that one was too long already.

Open source has been a huge part of my life for many years now. A friend once suggested I commission a song cycle about it, but I haven't been inspired in that particular direction yet. Reflecting back on my open source participation is interesting. I'll start with...

Launchpad

I'm one of the team leads on the Launchpad project. Launchpad's mission to connect open-source software developers within and between projects, and with Ubuntu packagers and users, is an amazing goal. Pragmatically, part of the goal is to help make Ubuntu the best Linux distribution available, and even the best general operating system available. That's a business goal. Beyond that, though, the engineers and managers behind Launchpad truly want it to be a part of improving the entire environment of open source. It's a huge, ambitious effort, and I'm proud to be a part of it.

Since I'm a manager at Launchpad, I have people to point to also.

• Leonard Richardson has already spoken for himself on his blog.
• Benji York doesn't have anything up on his blog yet as of this writing, but in addition to Launchpad, he has a long history of contributing to open source software. I think that Manuel and zope.testbrowser are highlights.
• As part of the QA team, Diogo Matsubara works on analysing the "oops" reports (error logs from the Launchpad application), bringing the prominent ones to the engineers' attention. In the analysis work, sometimes Matsubara suggests ways to fix the bug or finds the root cause, saving the LP engineers time. To accomplish that analysis, Matsubara maintains and improves the Oops Tools, a soon-to-be-open-sourced Django application that helps summarize the oops reports. Matsubara has also been working on improving the testability of Launchpad API clients. The first mission was to convert the Apport tests to avoid screen scraping and use launchpadlib. In a related effort, he's working on defining a API SLA, using the Apport work as the initial test case. Finally, Matsubara helps with process changes to make the Launchpad development easier, more fun and faster. He helped design and hack on the changes to make the recent "merge workflow" process plan work and rock for the LP team.
• Ursula Junque is a very valuable QA engineer for the Launchpad project who has initiated and contributed to many different parts of Launchpad. If she gives me a nice summary like Diogo's I will post it here. :-)
• Stuart Bishop, Launchpad's database administrator, is crucial for keeping Launchpad running smoothly. Beyond that, he has produced several open source projects. Perhaps his biggest contribution is his own timezone library, pytz, which to my knowledge is the standard Python library for this task.
• Maris Fogels hasn't posted anything yet either, but I know he has contributed to Windmill and done some great work on Launchpad's open source libraries.

Speaking of Launchpad's open source libraries, I should mention them too. None of them are mine, but I'm not sure anyone else will mention them all. If you head over to the LAZR project, you'll see many. I've heard mention of a few people finding uses for them, and I hope many more will too!

Zope

The Zope project was the biggest thing that brought me into the world of open source, and I learned a huge amount from it. It's been a big part of my life, and I wouldn't be the same engineer, or even the same person, without it. I think the changes that are happening in the Zope community that I talked about in the previous post are important and good, even if they are a bit painful to see sometimes.

I've contributed a lot to the Zope 3 project, and to the ZODB. Some of the projects I've done that are reasonably popular are zope.copy (over 25000 downloads, and actually useful in pure Python with only a dependency on zope.interface), zc.queue (over 19000 downloads), zc.relation (over 4000), zc.blist (890 downloads; heh, maybe I should move it out of "beta"), zc.async (only 355, but I know some use it heavily), and zc.catalog (meh, 267; I thought it was bigger than that). I have other, older and/or smaller ones too.

I've also contributed to the core projects, like ZODB, over the years as well.

Moreover, often while working on Zope projects, I've also made several other Python contributions. I'll mention those next.

Other projects

I just added the ability to use system Python packages with Buildout. That's a big deal for some.

I added the ability to have "latent slaves" to Buildbot. This means that, for instance, buildbot can start a slave on EC2 on demand when it needs to run a test. Hudson can do that now too, but at the time I added it, it was new, I think.

I did a bit of work on mechanize. The biggest job was teaching it to understand form labels, which made zope.testbrowser tests much more readable and usable.

I tweaked some table generation code in ReportLab.

It is fun to believe that I've contributed a decent amount to the world, and even (usually) had jobs while doing it.

Related work

Finally, since I'm here at a conference, I can't help but reminded that conferences--particularly presenting at conferences--can be very hard work too. I've presented several times (OSCON and DZUG most recently), and always about open source. I've also done crazy stuff in the past like writing a Zope 3 newsletter to let people in on all the changes when it was first being designed, and other promotional work for open source efforts.

In conclusion, though I don't share some of the more extreme opinions that some open source advocates espouse, it is a huge part of my life, and has been so for years. I hope I can keep working with and on it throughout the rest of my career. Maybe I'll even have the energy, time, and interest to work on it when I retire. :-)

Zope Summit, DZUG, repoze.bfg

I've spent a thought-provoking week in Germany, primarily in the company of people from the greater Zope community in Europe. By "the greater Zope community," I mean projects such as Plone, Zope 2/CMF, the Zope Toolkit, Grok, and BlueBream.

My week started in Halle, Germany, at Gocept's offices in a meeting called a "Zope Summit." 19 people gathered to try and figure out what we could collaborate on, how we could do it, and where things went from here. There was not a single answer to which everyone agreed. Some of the participants had heated words. By and large it was very friendly, though.

From my perspective, the only clear points of collaboration for us now are WSGI and libraries from the Zope Toolkit. Using WebOb would also give us more opportunities. Jim Fulton said he had began work on that integration over a year ago.

The most-discussed next step for people from the Zope community is repoze.bfg, also called BFG--a project with at least some measure of Zope heritage, but no direct representation in the meeting other than Charlie Clark's advocacy (see his EuroPython 2010 presentation and some of my previous blog posts).

In regards to BFG, I was pleasantly surprised to hear two pieces of news from Charlie. First, the next version of Pylons is slated to build on top of repoze.bfg. That's great news for the huge effort that Chris McDonough, BFG's lead developer, has put into this consolidation and the framework itself. It's also good news generally for sharing and collaboration in Python web frameworks: BFG, Pylons, and TurboGears will now all be interconnected, each building on the previous one, and all of them using WebOb and WSGI beneath.

Second, Tres Seaver, Chris McDonough, and others, are working on a layer above BFG for building content managing systems. While I strongly suspect that it would be years for this to compete feature-wise with longstanding efforts like Plone, a lighter-weight BFG-based CMS that could be customized much more easily would probably be immediately compelling for certain use cases.

After the Zope Summit, we moved Tuesday evening from Halle to Dresden, for the DZUG (German Zope User Group) conference starting the next day. The health of the German Zope community was startling to me. In America, the name "Zope" is often cause for many Python programmers to dismiss ideas and libraries, or look at them with prejudice. Here, developers acknowledge failings and are looking to the future, but the conference has a strength and positiveness that is a stark contrast to the attitude to which I have become accustomed.

That said, the conference, like the summit, was also a time of evaluation of where the Zope community is now, and where it might go soon. BFG was again a central topic in the meeting rooms and over meals, and the subject of a couple of presentations. BFG got positive notice, though some expressed a desire to see a clear path for more functionality, such as user management.

The first conference day started with Jim Fulton, the original Zope leader (or "Zope Pope," as he was called) talking a few minutes about his perspective on where Zope came from and where it is now. I interpreted it as a farewell, and certainly a repetition of the fact that he is no longer the Zope leader. Beyond that, he talked about mistakes made in the Zope project, and strengths and lessons to keep in mind for the future. A central opinion he stated was that Zope's problems came in part because it tried to be two things at once: a CMS and a framework for Python developers. He advised against making that kind of mistake in the future.

That evening I presented a keynote titled "Launchpad and Zope." It was a set of five loosely connected lightning talks, using a clever PyGame program fellow-Canonicalite Benji York wrote. It was very well received, and I had a lot of fun. Putting the slides up might be a little tricky because of the custom format, with movie integration, but maybe I'll try to put them up. These were the topics, with a short summary for each.

Launchpad and Zope

The Launchpad project aims to connect the open source community together, and particularly to connect Ubuntu developers, packagers, and users within it. It uses several key Zope Toolkit libraries, especially templating, publishing, security, interfaces, and the component architecture.

Launchpad's webservice

Launchpad's webservice is easy to use and exposes an amazing amount of Launchpad's functionality.

Open Source Libraries

Launchpad has a number of open-sourced libraries, such as lazr.restful, wadllib, lazr.resftfulclient, lazr.config, lazr.batchnavigator, lazr.uri, and lazr.delegates.

Buildout and System Pythons

The Buildout 1.5.X line introduced some nice features to make Buildout usable with exposed system Python libraries.

Launchpad and Zope?

Launchpad's use of Zope has some downsides and some upsides. We plan to keep some libraries because of their usefulness, but would like to remove others, such as Zope's publisher, because of complexity.

The other presentations were pretty much all in German, and I don't understand German well enough for a technical talk. One other was in English, but was about Plone, with which I have little practical connection or interest other than wishing Plone well.

While I was at the conference and the summit, I also spent time discussing interesting topics with attendees in the shared areas. Here are some representative summaries.

Chameleon

Launchpad is interested in using Chameleon, and had some work done over a year ago in that direction. I talked about some of the problems I had encountered and got some hints on next steps. I also learned that it was not yet in production for one of the people promoting it; and got confirmation that the maintainer is working on a rewrite. These last two items concerned me.

zope.security

Volker Jaenisch brought up some concerns about the zope.security package. After discussing with him, we ended up with these observations and recommendations.

• emphasize in zope.security that __parent__ *must* be set in order for expected behavior to occur. (Recommendation: use a factory?)
• z3c.form: document that security is not ready until after addition.
• zope.security: design a cache to be held over multiple requests? Can be a big performance win, but must be very careful.
• zope.security: document that cache has potential edge cases
  • as long as cache is maintained, moving an object needs to either completely or partially clear the cache
  • similarly, children of moved objects must have their cached security values invalidated
  • would be nice to have a policy that did not have cache
• zope.security: would be nice if invalidations after grants and other security changes were localized (currently global)
• zope.security: would be better if the test showed integration between the policy and the proxy unauthorized

ZODB/ZEORAID

Christian Theune and I talked about the ZODB for a while, and he described some of the advantages and upcoming work on ZEORAID.

Django DB connections

Marc-André Lemburg showed me some interesting code he has to generate code to interact with the database, and we looked into how Django controls its Database connections.

It's been a nice week. The people at the conference and the summit were great to be around, and I particularly appreciate all the work from Gocept to organize these events. I'm also honored to have been invited as a keynote speaker.

And I'll be glad to be home. :-)

Zope Interfaces and Python Abstract Base Classes

Jim Fulton (the architect of Zope 2, Zope 3, zc.buildout, and so on) and I talked briefly on the first day of the sprints about whether we could take advantage of the Python 2.6/3.0 Abstract Base Classes (ABCs), either replacing Zope interfaces or somehow using them to our advantage.

So I don't misrepresent him, I'll state my conclusions. Ask Jim for his.

Unfortunately, I came to the conclusion that ABCs, as of their current implementation, are anemic and largely uninteresting from zope.component's perspective. Some points:

• The basic ABC mechanism simply allows hooks into isinstance and issubclass, so that you can ask, for instance, ``isinstance(x, Y)``, and ``Y.__instancecheck__(x)`` is consulted, if the method exists. Jim said we could make zope.interface interfaces implement these hooks. That didn't seem very interesting to me: to me, ``isinstance(x, Y)`` is a very different semantic question than ``Y.providedBy(x)`` and the difference is valuable. Same for ``issubclass(X, Y)`` versus ``Y.implementedBy(X)``. I'd rather not have zope.interface muddy that water.
• The abc module allows you to create abstract base classes. To declare that a class is a concrete implementation of the ABC, you can either have the concrete class subclass the ABC; or, you can call the ABC's ``register`` method. In the current implementation, the registration is one-way, stored on the ABC. Therefore, you can ask an ABC, via various internals, what its "subclasses" are; but you cannot ask a class or instance about the ABCs it subclasses or is an instance of. zope.component needs to be able to ask what a class implements, or what an instance provides.
• The zope.interface implementation is highly optimized, and relies on cacheing results for speed. The abc module is not geared for speed.
• I wondered if we could at least leverage the default Python ABCs, describing mappings and sequences, to create interfaces from them, deprecating the ones in zope.interface.common. Jim seemed disinterested in that, saying that he hadn't found ours useful. I have found them useful--as a shortcut to describing a contract based on a mapping or a sequence, usually--so that still seems like a possible win to me.

Jim pointed out that zope.interface scribbled its metadata on classes, which some people found distasteful, and at least the abc module didn't do that. This characteristic of ABCs might please some people. A global data structure might be another way for zope.interface to do its work while answering that concern, if that ever became something we wanted to solve.

So in conclusion, the basic mechanism and the module were not really of interest to me; I felt that the library of ABCs might at least be interesting, but Jim disagreed.

Too bad. But hey, maybe we can throw out the Zope testrunner in favor of nose! :-) Jim proposed that idea, and thought it might be doable, with a little extension trick to make nose support Zope layers....

Back from PyCon 2009...a week ago.

I came back about a week ago from PyCon 2009...and went straight into getting our house ready to offer for sale, and then performing Anima Mundi by Mark Scearce in Raleigh, NC with my wife, Karyn, and some old and new friends. Good stuff, but exhausting.

I have quite a few notes from PyCon. My first mission there was to announce some of Launchpad's recent open-source work, in particular lazr.restful. I am excited about that, and have things to say, but I'm going to wait to blog on that just a bit longer. The docs need some work, at the least.

While, of course, the Django contingent at PyCon was very large, I was pleasantly surprised that the Zope/Plone community had a good showing. There were a few talks from the community, and BoFs included "I'm not ashamed to be a Zope programmer," "I love Zope," "I hate Zope" (the same crowd attended both the love and hate variants, I'm told), and a generic "Zope" BoF. The sprinting was enthusiastic and lively, and some cross pollination with some of the other non-Django frameworks also added some excitement and interest. The huge international Plone community using and sharing generic Zope libraries also has increased energy behind the Zope libraries.

Perhaps the most interesting generic Zope conversation I had was an attempt to identify what unifies the "Zope" projects. Thanks to efforts to bring Zope 3 libraries to Zope 2 and Plone, there is arguably more of a common theme across projects than in the past. There was some consensus that the following two ideas unify Zope projects.

• Zope provides an unusual degree of low-level pluggability and interchangeability thanks to a contract-based component system.
• Zope usually used a graph traversal approach to converting URLs to code, which has an advantage in that it is arguably easier to convert code back to URLs (walk back up the graph) than other approaches that do not have a natural reciprocal.

(For thoughts on the second point, see my old blog post.)

I have several specific PyCon reports that I'll post separately. All in all, I very much enjoyed the conference, though much more for the conversations and the sprints than the talks.

Speaking at OSCON 2009: Launchpad Foundations

My speaking proposal for OSCON 2009 was accepted: "Launchpad Foundations: Learning to Leverage a Component Architecture". Here's the quick blurb:

Study gains and losses in how Launchpad, a collaboration web service for the open-source community, used a Python component library from Zope 3 to help manage a large project. Discuss when the approach might be appropriate. Code examples include automatic REST web service generation. Demonstrate how the component architecture might be leveraged in popular frameworks such as Django.

Go to the details page for the full blurb. It should be interesting and fun to prepare. I'm interested in seeing if I can fit in references to some of the work that the pypefitters guys have been doing as well. WSGI plus zope.component seem like a pretty good pluggable base combination for web applications to me.

Getting VMWare Faster

I like Macs, and OS X. I like Ubuntu too, and with my job at Canonical, I develop with it.

I've been developing in Ubuntu in a VMWare Fusion image, for a variety of reasons. I recently got really tired of the slow speed I was experiencing, though. I decided to investigate what I could do to keep my Mac/Linux story portable, but faster, that didn't involve a new computer.

Random Googling to the rescue!

The most persistent advice I read was to get an external disk. I also read that using a reasonably fast external disk over a relatively slow connection like USB 2 might not do so well. I also read...a whole bunch of other things.

Here's what I did:

• I already was using my FireWire 800 port for some other drives, so I decided to go for broke with an eSATA Express Card adapter.
• I got a 500GB 5200RPM 2.5" disk drive in a rocketfish enclosure with eSATA and USB connections.
• After connecting everything up, I built a new VMWare image on the external drive. I used Ubuntu 64 bit, because I read that VMWare could take advantage of some 64-bit opcodes. The 64bit Ubuntu image is named "AMD64" but I read that it works fine on Intel 64 bit. My experience bears this out.
• I gave the image 1.5G of my 3G RAM.
• I configured the image to pre-allocate the necessary space on the hard-drive because VMWare Fusion mentions that this might speed things up.
• I made sure that the 3D graphics acceleration was not turned on for the image, since Linux can't use it anyway.

The results were gratifying! The image certainly feels snappier. More concretely, running a (presumably representative) subset of the Launchpad test suite took Between 20 and 30 minutes on the old image, and between two and three minutes on the new one. <happy sigh>

Now the next question would be "which changes made the biggest difference?" I'd love to know--but probably not enough to actually do the experiment. Instead, I'll use the faster speed to help in this coming week's sprint to abstract a REST webservice framework!

Oh, the farmer and the cowman should be friends: URI parsing with Routes versus graph traversal

(This post's title alludes to a song from the musical Oklahoma, in case you were wondering.)

I, like many web application developers, am impressed with the Routes model for mapping a URI to application code (as in RoR, or any number of Python versions). I plan to use it for "hobby" work, and I'm advocating it at my job.

For many web applications, it seems to work as well or better than the other approach to web application URI parsing with which I'm familiar, graph traversal. In the graph traversal approach I know, you typically divide up the URI path elements by slashes into individual path elements. For example, "/musical_theater/rodgers_and_hammerstein/oklahoma" becomes ["", "musical_theater", "rodgers_and_hammerstein", "oklahoma"]). Then you start with a given graph node and use each path element as input to traverse the graph. For instance, repoze.bfg strictly uses __getitem__ to traverse the graph, so the example URI above might equate to root_object["musical_theater"]["rodgers_and_hammerstein"]["oklahoma"].

The Routes model is particularly nice for web sites publishing square, non-hierarchical data. If you don't have a graph to traverse, then you have to do something else!

Moreover, I buy into the argument that Routes encourages you to think about your URI space separately from your model. This fits in well with REST philosophies, in particular if you regard your URIs as a significant aspect of your user interface.

In defense of graph traversal, I generally have found that traversing model objects has resulted in reasonable URIs. Also, one could traverse a graph of abstract traversal controllers instead of models (and in fact, at my job, that is what the code of Launchpad does, as of this writing).

But typically, graph traversal does tend to mix model and URI in a way that can force "model" objects into a system when all you really want is a URI.

For instance, in Zope sites that I have designed, I have frequently felt awkward about the top-level design--the part of the design in which you are arranging top-level access to your models. This part of the website functionality often does not map naturally to model objects. In Zope using the ZODB, the nodes in the traversed graph are usually persistent objects, and so the top-level objects have a "model" feel; and yet they are usually just scaffolding until you get into the meat--the real models--of the application.

As another example, URIs in which path elements are really query-string-like filters on a view rather than true graph traversal are possible, but not as natural with graph traversal systems. For example, consider this URI from trulia.com: http://www.trulia.com/for_sale/3p_beds/2p_baths/SINGLE-FAMILY_HOME_type/resale,new_homes_lt/38.652833,38.976488,-85.838055,-85.455951_xy/10_zm/. (No, I'm not planning on moving to Indiana.) That URI reads well, and follows typical REST advice to move information into the URI. It's doable with graph traversal approaches, but is not really traversing a graph.

Graph traversal has some strengths as well, though.

An obvious one is when you have a graph to traverse. Perhaps you have a CMS in which documents can be arranged into arbitrarily nested folders. Or perhaps you have some concept of "projects" that can contain other projects, to an arbitrary depth.

Of course, in the same way that graph traversal can be made to handle pure-URI stories, such as with Launchpad's abstract traversal controllers, Routes can handle graph traversal. But I argue that graph traversal is more natural to, um, traversing graphs.

In particular, if you have graph nodes that can be dynamically created that have different traversal rules, as in the CMS example above, then defining how to traverse per graph node can be more natural and cleaner than specifying the rules in a routes file and a single controller.

Also, when a routes system starts to make heavy use of regular expressions--say, a rule that specifies anything beyond static strings, a controller, an id, a view, and a "catch all" for the rest of the URI--simple graph traversal approaches can be much easier to express and understand. (Examples of relatively simple traversal approaches are the Launchpad navigation traversers, or the repoze.bfg __getitem__ approach.)

So, they both have applicability. Maybe we can combine the two approaches when it makes sense. The farmer and the cowman should be friends. (You get to decide which approach is the farmer, and is which is the cowboy, though see the postscript.)

For some projects, Routes or graph traversal alone might fit the bill perfectly. I do tend to guess that Routes is the better general-purpose approach. But for some applications--if they present a complex data structure, for instance, and especially one in which one or more aspects of the site can be presented as a graph--then maybe you ought to have Routes for the top of your site, which then can defer to graph traversal for certain parts of your site that make sense.

megrok.trails goes down this road, but not quite the way I'm thinking of at the moment. It fits Routes-style traversal within a larger context of graph traversal. I'd like to turn that inside out: when appropriate, have a Routes mapping with a wildcard that consumes the entire tail end of a URI, and then sends this to an intermediate controller, which uses graph traversal on the wildcard part of the URI to find the "real" controller. Routes is entirely in charge initially, and explicitly defers to graph traversal if so requested.

I wouldn't be surprised to learn if such a thing existed for Routes. It would be pretty easy to code up. I'd like to use something like it.

---

Postscript: For what it's worth, I'm struck by an overwhelming desire to relate the farmer, making fences, to Routes, making nice, simple URI rules; and to relate the cowman, herding free-range cattle, to graph traversal, letting you walk over arbitrary model graphs. But metaphors like that sometimes get people up in arms, because the Routes people might want to be the rough-and-tumble cowboys, and the graph traversal people might want to be the practical and pragmatic farmers. So forget I said anything like that.)