My Claim: "MTV" Is Silly.

repoze.bfg's creator, Chris McDonough, in an informative and corrective comment to my last blog post, among other things disagreed with me about my assertion that "repoze.bfg doesn't provide a model story; it provides a traversal story."

I'm afraid I didn't communicate myself well enough. He might still disagree, and I might still fail to make my points clearly, but I found this the most interesting observation I made, so let me try again.

• I think applying Model-View-Controller to client-side application frameworks makes sense. MVC actually makes sense, with a reasonably clear delineation of responsibilities, when you look at Cocoa, for instance, or even in some of the more recent JS frameworks (Sproutcore, for instance).
• All of the reasonably recent web frameworks out there with which I am currently even vaguely familiar (Zope 3, Django, RoR, bfg, Pylons) have a model and something responsible for rendering. The rendering usually goes out to a template, but not always. It's not really cooked that you have to, and usually that's regarded as an advantage and a flexibility. "Render however you want! Use whatever library you want!" So, the heart of the system is Model-View. The Controller isn't there, and the Template is an implementation detail of the View. "Model-Template-View" or "MTV" just seems silly to me. I'd prefer it if everyone acknowledged that our web frameworks usually just have "MV," and move on.
• bfg doesn't care what it is traversing. Pretty much, give it things that it can use the __getitem__ protocol on, and then when it has consumed the path, it'll adapt to a view class. The __getitem__ bits could be a model...or not! What if your data model didn't jibe with your URL model? That's completely reasonable, and the routes guys have plenty of examples in their apps because of how they think about URLs. What if you like the __getitem__ pattern for your URLs, but your URL story is different than your model. You might build a true MVC system with bfg: pure data-driven models; the bfg traversal system used exclusively over "controller" objects that handle traversal and maybe request (i.e., form) parsing; and views that adapt the controllers to *only* render. Maybe the controllers even optionally have WADL-like contracts based on request inputs.

So, my point was actually not that bfg was cheating in any way--certainly, for instance, nothing like Zope 2. To recap, then.

1. I find the "MTV" term to be specious generally, whatever the web framework. That's just a criticism of the term, as web framework marketing has adopted it in the past few years. I'd love for it to retire.
2. interestingly, I don't think bfg is truly tied to the "MTV" model. It doesn't care what it traverses. The MTV model works fine, but a story like what I described, in which the models are maintained separately from the URL space, and the traversed objects are traversal "controllers," would also work well. Then thinking about any additional responsibility of the traversed objects is an interesting exercise, especially in light of REST-ian approaches.

So, yes, Chris is right, from one perspective, bfg is as much MTV as anybody else. That's fine. I'm just railing against the term, and saying that bfg can be used for more than just "MTV".

Repoze.bfg

As mentioned earlier, I've spent time looking at other frameworks lately.

I've spent more time on Chris McDonough's repoze.bfg than any of the others so far. This is probably because, as discussed below, it's very minimal. It's also documented well. Finally, it follows a few standard old Zope patterns that don't require much thought for me to process. Given all that, I can understand it quickly, and so am enticed to spend more time to read and think a bit about its design.

I looked at it again because of this recent bit of marketing: http://plope.com/whats_your_web_framework_doing. Chris makes his point, which is valid; and he's selling to his design's goals, which is the point of this kind of presentation.

As an aside, I'm a "can't we all just get along" kind of guy, so the fact that the trade-offs of repoze.bfg's design are not discussed in comparison with the other frameworks bothers me, even though having done so would have made the piece much worse marketing and much harder-to-read communication. (A repoze.bfg design tradeoff example: if you always need authentication and authorization for your web apps, you'll need to plug in and understand more WSGI middleware, and then the given comparison is not as pertinent, at least for Grok and Django.) I actually wouldn't be surprised if repoze.bfg still would do very well in the chosen metric, if the set up actually did include authorization and authentication middleware in the profile; and if it didn't discard the webob.Response. That would have been mildly more interesting to me. But, whatever, it's marketing, and I get Chris' point.

The name of the framework, "bfg" is funny on multiple levels. The level that sticks with me is that the F[*&^%$#] G[un] is really not that B[ig]. As the documentation points out, this is a very minimal framework:

Minimalism: repoze.bfg provides only the very basics: URL to code mapping, templating, and security. There is not much more to the framework than these pieces: you are expected to provide the rest.

That's nice for a "pay for what you eat" story, as the documentation says elsewhere. But it's also insufficient for any website I've ever made. There are at least some suggested patterns to follow elsewhere within the repoze meta-project: repoze.who and repoze.what are available for authentication and authorization, for instance.

But it is a framework that wants more guidance, more "rails," more framework, to get some basics done. What about web form helpers: maybe we ought to use Ian Bicking's stuff. Or what about REST helpers? You might be able to write some interesting adapters from a generic RESTful view to a CRUD-ish interface, like the patterns I've seen in Rails. But it's not there now (and the documentation states that it is an active goal to hide the zope component architecture, which could have helped with this).

While it's nice to be lightweight, I think this would make a more appealing sales pitch. Maybe it's in the plans to build related libraries and integrate them in "building with repoze.bfg" tutorials, or maybe it's antithetical to Chris' goals, who knows.

Of the three features that the framework provides, the view and templating story is the least interesting to me. It seems very similar to the Zope 3/Grok story. I intend to checkout Ian Bicking's webob library, and I hope to use chameleon at work and for hobby projects, but that's the extent of it.

The security story is very similar to Grok's. They both forego framework-level security checks during traversal, I believe, while they differ in the last step: repoze.bfg security-protects the last traversed object within the view code, as I understand it, while Grok security-protects the view. For what it is worth, I prefer the repoze.bfg approach.

The traversal story is the most interesting to me. When I first heard the repoze.bfg traversal plan of "__getitem__ over the model, period," as opposed to the more flexible standard traversal story of Grok/Zope 3, I was skeptical, but the more I think about it the more I like it. The traversal story in Zope 3 has always been a pain to use for me, and while there might be a more powerfully flexible way to alleviate that pain than the repoze.bfg approach (and I think Grok might have tackled this already), the __getitem__ simplicity still is appealing.

In that vein, I find that the assertion that the repoze.bfg code is not MVC but "MVT" (Model-View-Template) like Django doesn't feel right. repoze.bfg doesn't provide a model story; it provides a traversal story. As such, you could be traversing over models or controllers; the code doesn't care which. This is "TV" (Traversable-View); or "[MC]V," from a regex perspective on MVC; or some other odd acronym. Not MVT.

In any case, while I might explore using repoze.bfg on some hobby projects, primarily to get my hands dirty with WSGI and get a better handle on a couple of Ian Bicking's libraries, I won't be working with this at work, and I have a higher personal priority to get some time with Django. I'll probably continue to follow repoze.bfg's development from a distance for now...and be glad that I don't have to write any marketing myself.

Interactive Fiction, Declarative Domain Specific Languages, and Web Frameworks: Introduction

This is the first part--just the introduction, really--to what I intend to be a small series.

I've liked interactive fiction pretty well since I was a child. If "interactive fiction" doesn't mean anything to you, you might be familiar with the old Infocom games of the 1980s: Zork, Planetfall, The Hitchhiker's Guide to the Galaxy, Leather Goddesses of Phobos, and so on. That's interactive fiction.

Even though interactive fiction hasn't had much of a commercial impact since the 1980s, the genre has been alive and well, and actually doing amazing things, ever since. Shareware and freeware drive it now.

If you're willing to give it a try, Emily Short is considered one of the current masters. Galatea, for instance, might be an eye-opener: free, only 10 minutes long to play, and not a puzzle per se, but pretty amazing. Give it a try. You also might enjoy checking out the experience of blueful. I intend to try out Lost Pig with my five year old, playing it with him. There are also plenty of long fantasy, sci-fi and mystery games out there to be found, still usually free; and free software to play the games for Windows, Mac, Linux, Palm, iPhone (search the Appstore for Frotz), and plenty of others. This is really cool stuff.

So how do you write interactive fiction? Well, that's cool too. What's going on there right now fascinates me, both because it inspires me to want to give it a try, and because it has a different perspective on similar issues that I've been observing with web frameworks.

Later in this series: Inform 7 versus TADS, domain specific languages and design, pondering declarative goal-directed programming for web applications, RoR and Hobo, and various other things that I will make up as I go along.

Buildbot and Amazon Web Services Elastic Compute Cloud ("AWS EC2")

If you might care about Buildbot, you probably already know about it: "a system to automate the compile/test cycle required by most software projects to validate code changes," as the site puts it. It's written in Python with the Twisted framework.

At Canonical, I've been doing some work with it, and, lately, on it. My first accepted git branch added some bzr helpers. That was cool, but small. The one that was accepted this week is a bit more interesting, I think.

I've made it possible to hook up one or more on-demand, or "latent" slaves to the buildbot master. In the current single concrete implementation of the abstract class, for example, a latent slave always claims to be connected and ready to perform builds, even though it does not really have a remote connection to a machine ready to do work. But when a latent slave gets a build request, it instantiates an Amazon Web Services ("AWS") Elastic Compute Cloud ("EC2") virtual machine, using the image (and therefore operating system) of your choice to run the tests. It uses the nice AWS Python library, boto, to make the magic happen.

Implementing the same thing for other similar cloud computing services should be pretty easy. The buildslave.py module now has an AbstractLatentBuildSlave class. All you have to do to is subclass that and implement one method to start a virtual machine, and one to stop it. It's described in the pertinent section of the documentation.

I figure the new feature is probably only interesting for a relatively small subset of buildbot users. But it's still pretty cool, and a significantly different variation on what was there before. This is supposed to be released in 0.7.10, which is slated to be RSN, as I understand it.

If you are curious enough about it to want to poke around, the git master branch is here. You can find the EC2 latent slave in buildbot/ec2buildslave.py. The documentation also has extensive additions to try to help you get started.

Looks like I'll be doing a bit more buildbot work in February. Cool.

Catch-up

I feel like I've been playing a whole lot of catch-up lately.

Mark Ramm's September '08 blog posts about how Django could learn from Zope 2's mistakes made one point (actually from the second post) that struck home strongly: more innovation happens elsewhere than within one given community.  You have to pay attention to it and be a part of it.

I have been a part of some cool, and uniquely valuable, stuff working on and with Zope 3.  I've also tried to keep abreast of new technologies, studying Dojo, and Programming Collective Intelligence, and SproutCore, and Objective C, and so on.  Notice that another web framework is not in that list.  Maybe that's because I was working for a company, Zope Corporation, in which working with other frameworks was not really an option (not that I minded, mind you; I like Zope 3).

But employer policies and positions are not really a good excuse.  I should have been actively studying the other web frameworks anyway.

Now that I'm in a work environment with more opportunity for cross-pollination (working for Canonical) I feel like I'm trying to swim out of a backwater to catch up with the rest of the web developer world.  It's daunting and stimulating.  Ideally I can find a way to integrate the best of my past with what the rest of the world is doing.  I've been a part of some innovation too, and I believe some chunk of it is worth bringing forward.  But I need to catch up.

I've been saving up my notes on REST while I read Leonard Richardson's excellent O'Reilly book about it, hopefully for a series of posts.  It also is an interesting, if somewhat dated, view into the world of Ruby on Rails.  In the alt-Zope world, I've been looking into what Tres Seaver, Chris McDonough and friends have been doing with repoze, especially repoze.bfg.  (I've already been somewhat familiar with Grok, but that's so close to Zope 3 that studying it really doesn't go too far in the way of cross-pollination.)  Obviously I need to spend some quality time with Django (I've done just a bit so far) and I'm impressed enough with Mark Ramm's presentations that I figure I ought to spend some time with TurboGears.

Like I said, daunting. And stimulating.

But meanwhile...I've also been looking at what interactive fiction has been up to since the last time I looked!  And that's what I intend to blog about next: the declarative domain specific language in Inform 7, and maybe how it relates to this crazy web developer biz.

Learning How to Blog

It's been awhile--over two years--since I blogged. The reasons are simple: I'm busy with work and family, and perhaps I associated blogging too much with professional or scholarly publishing. The second point had two corollaries: my posts were often too big, both in terms of engaging readers and finding time to write; and I felt insecure and shy about saying much. I'm still busy, and I still want my posts to have some value, but I'm hoping I can get back to blogging. Why?

• I have stuff to say!
• I'm working from home now for a distributed company (Canonical), so blogging is a way to share (non-proprietary) ideas in the company.
• Twitter and Facebook have shown me lately the value of making connections with distributed friends and acquaintances.
• It's a reason, and a way, to work out ideas gradually.
• I now understand blogging as being (at least potentially) more social--more of a conversation than the higher stakes of a formal presentation or publication.
• Assuming I don't perform the blogging equivalent of stripping in front of a religious monument, it is conceivably good publicity, if that ever helps. Or, maybe, there's no such thing as bad press? In any case, my boss wants me to take a more public position in the open-source community, in part because I'm driving some of the open-sourcing work for my team. (BTW, I may be going to PyCon! See anybody I know there?)

What do I have to say? Well, off the top of my head:

• I'm working with the primary author of the O'Reilly REST book, and I have some nascent thoughts about REST and what I'm learning from him that I'd like to work out.
• I presented a speech at a company conference this past October about my take on why and for whom the Zope interface and component libraries might matter. It would be nice to make that a bit more general and share it.
• As I said, I'm working from home now, and I really enjoy it. One reason I enjoy it is the processes that my company, and my project, have for helping a distributed team work together. I'd like to share them, because I think the whole experience is great.
• I've done some more open-source work lately, ranging from adding the capability for buildbot to launch AWS-EC2-based build slaves on demand, through adding support for the bzr revision control system to buildbot, through adding cookie helpers to testbrowser, to packaging and releasing some software developed by other folks in my company. I'd like to talk about them a bit.

So, there's plenty of reasons to blog. How can I make my blogging experience better? What can I learn from the last time I tried this?

• As I said before, treat this as a conversation. I usually don't have time to prepare something that approaches being authoritative, so admit it, move on, and be ready to listen.
• That said, remember ye olde paper-writing days from school: outlines RULE. Write an outline, at least an informal one, first.
• If the outline starts to look big, do it in broad-brush and DIVIDE IT UP across several posts! The outline will help keep the posts make sense together, and focusing on one or two bullet points per post will keep the blog entries shorter and less imposing for readers, and will be a smaller, easier-to-schedule block of time for me to write.
• Let my light(ness) shine! I have occasionally been reasonably amusing in my life, and, as a reader, I find that posts with a bit of lightness and levity are easier for me to read. Being lighthearted probably puts me in a better frame of mind when I write, too. "A spoonful of sugar helps the medicine go down"?

So, that's the current theory, anyway, for teaching myself how to blog. Let's see how it goes.

Introducing Zope

I'm going to be introducing someone to Zope 3 starting tomorrow. The goal is to build a productive long-term contributor. I'll be investing a decent amount of time, and also be expecting a big investment of energy from the student.

To start off, I want to answer this question: what is Zope 3, and why do we use it?

Amazingly, one of the community's problems is that we don't agree on an answer. Steve Alexander gave a reasonable definition in a EuroPython 2004 talk, which Philipp von Weitershausen quotes in the first chapter of his book. Here's what Steve said:

Zope is

• a collection of free software
• jointly developed by Zope Corporation and a large community of software developers
• that you can use in whole or in part
• to manage complexity in gluing software components together,
• securely publish objects on the web and other systems,
• and make it easy to do Quality Assurance.

I'd agree with that, though I think the word "Python" belongs in there somewhere, and it's hardly an exciting marketing blurb for newcomers familiar with the buzz from other web frameworks. I guess I'll start with Steve's answer, but then I need to dig in.

• The "glue" Zope 3 offers is by working with contracts--interfaces--and providing a couple of lookup mechanisms based on the contracts--adapters and utilities. I'll delve into that a bit, and give some examples at the Python prompt.
• The "quality assurance" typically comes from doc tests of various types. I'll delve into that a bit, and then encourage writing some modules with it. Hopefully together we can come up with an interesting example package to work on, or else I'll have to come up with a random one.

Now we start veering away from Steve's definition--or perhaps veering into some of the details.

• Our persistence mechanism, at least at Zope Corporation, is almost always the ZODB. It is an essential part of becoming a productive long-term contributor at our company, so I'll delve into some details here, including talking about transactions and conflicts. We'll make the package from the previous step ZODB-aware.
• We can talk about events as a concrete example of our "adapter" lookup concept (as an aside, adapter may be a misnomer, and the evidence may be that it is not a helpful word to explain many of the uses to newcomers). We can probably make our package generate some reasonable events.
• Catalog integration is an obvious next step from a package perspective, and I regard the key reference and intid knowledge that would be part of the catalog discussion to be essential to expert use of the ZODB.
• Authentication and security touch on another of Steve's points and also are a concrete example of utilities. We'll want to delve into that.
• And finally, we have the web. The Zope object publisher, and specifically the Zope request is a core concept. Getting into views will highlight our flexibility but also our holes in comparison to other packages out there (as an aside, I wonder if we as a company or as a community can steal the widget work from some of the other frameworks, like the TurboGears ToscaWidgets?) We'll return to quality assurance with testbrowser doc tests to interact with the publisher and our views. And we'll want to dig into authentication and security again.

I like this, but I'm also leery of my predisposition to teach with too little initial hands-on excitement. I'll try to brainstorm some initial "wow" experience tonight and tomorrow morning. Maybe I'll split up my time between back-to-front teaching, as I describe here, and front-to-back teaching, as is popular with recent "write a blog in ten minutes" web frameworks. I do think Zope stands out from the crowd more when viewed back-to-front, at the moment.