Out on the edge with the ZODB

Someday I'd love to write a book called something like Application Development with the Zope Object Database (ZODB). Writing it would let me share the information I've learned, as well as make me learn the bits I still haven't gotten around to. That's how I really think of my Zope 3 work--in that context, I'm

1. A Python programmer
2. who uses the ZODB and related libraries
3. with web-related components and libraries.

The ZODB is a very nice tool to work with. The Zope books out there just touch on it, and I think many Zope technologies can be introduced from the perspective of the ZODB, rather than the other way around. Maybe I'll write that book yet.

In any case, last week at work we found another interesting ZODB edge case to contemplate. More than a year ago, a colleague had added an interesting and useful feature to a package I largely created, zc.catalog (PyPI, browse SVN, checkout SVN). Rather than processing catalog requests as they came in, the feature allowed the requests to be queued for the end of the transaction. The big win here is for objects that may have multiple changes within a transaction, such as during object creation and initialization: we can collapse all the index requests for a given object into one, and hopefully save redundant work.

When I studied how this this had been done, I saw that the work queue was a persistent object with a connection to the database. Why, I wondered? This object should always begin and end every transaction empty! Can't we just use a purely transient object, discarded at the end of the transaction?

The answer (or at least my answer) was subtransactions. If you index something, then start a subtransaction, then index something else, then abort the subtransaction, then the first index request should remain in the queue. We really want transactional behavior, not persistence, and persistence is the easiest way to get that (without writing a transaction manager and so on).

Our problem is somewhat obvious in retrospect: using a persistent object for this queue can cause write conflicts. In the case of an object without special conflict resolution code, concurrent transactions that both cause the catalog to index anything will make the queue "dirty" in both transactions, and thus generate a conflict error. The object must have a specific conflict resolution policy that notices when all three states (mine, the conflicting one, and the shared historical one) are the same; or (I learned) you can have the client code call _p_invalidate on the queue after it is empty to discard the dirty state and return to the initial version.

The queue was a persistent dict. BTree buckets have conflict resolution code, but unfortunately not this bit of logic, so they would not have helped (if I remember and understand correctly, an empty bucket usually means that it should be discarded from a tree, so it can only resolve a conflict safely if all three states are empty, not just the two conflicting ones). The remaining solutions are to write an object with the conflict resolution code, to use _p_invalidate, or possibly to make a custom transaction manager. As I understand it, a colleague plans to choose _p_invalidate, as well as to add tests for the queueing behavior that were omitted from the initial addition.

The transaction manager seems like it might be a "purer" solution, but the other approaches might be more practical. In any case, I found the problem and resolution to be fascinating, and it added another few bits of ZODB knowledge to my store.