Clojure/conj: The Questions and the Answers

Wow. I just had a full immersion experience into the Clojure language and community, and it was awesome.  I'll write about in two posts: questions I tried to answer, and notes from the presentations.  Here's the first.

Thanks to my wife, my boss, and my employer, Canonical, I got to attend the Clojure/conj conference here in Raleigh, as well as the training session beforehand.  It was a full and exhausting six days, and I didn't even participate in the nighttime events because I went home to be with my family.  I left the conference with knowledge, excitement, and inspiration.

When Canonical gives employees time to go to a conference, we have to summarize it.  The summaries are often company-internal emails, but I like blogging about them.  So here we are.

As I said, I'm going to tackle the conference experience in two posts.  For this one, I spent a lot of time asking questions.  I've been looking at Clojure for over a year now, reading various books and messing around in spare moments.  I accumulated a lot of questions about the language and its community, and I had even more while I was being trained and while I was an the conference.  Here are the questions I had, and the answers I came away with.  I'll be appreciative of comments, emails or tweets (@garyposter) if anyone wants to correct or add to my answers!

While most of the questions are those that I asked others, I'll start with three questions that people asked me.

Why Clojure?
I've been interested in finding another language to seriously explore other than Python for several years.  Beyond some kind of intuitive and minimal threshold of practicality, I wanted a combination of two things:

• a syntax and readability that appealed to me subjectively, and
• a significant difference from Python that stretched my mind.

Some languages, like Ruby, didn't meet either criteria.  Some, like Erlang, didn't appeal to me syntactically.  Some, like Go, didn't stretch my mind enough to be appealing enough to invest in.  If my job asked me to dig into one of those, I'd enjoy it, but for self-study and hobby work, they didn't make the cut.  Clojure does.  I continue to enjoy Clojure and feel challenged by it as I gain more experience with it.

Clojure also has a mix of agility and rigor that appeals to me.  The community draws directly from both Ruby and Java worlds, and I find the mix to be fascinating and compelling.

Choosing a language for a project or a team would have very different goals than the ones I have here.  That said, Clojure also feels like a practical and effective choice for many problem domains, and that's certainly the language's goal.  On the basis of what I know so far, I'd definitely consider it for that kind of discussion as well.

How was the training session?
Excellent.  The three trainers, Aaron Bedra, Stuart Halloway and Stuart Sierra, all work for Clojure sponsor Relevance, are members of the core Clojure development team, and are authors of some of the Clojure books.  They moved quickly over the three days, using the open-source labrepl for hands-on work.  I was basically familiar with almost all of the written material--with the notable exceptions of a very cool ClojureScript demo from Brenton Ashworth and a development-to-deployment demo and discussion that Aaron gave, extracted from the upcoming second edition of Programming Clojure.  However, the verbal presentation went into much more depth, and I tried to take full advantage of the opportunity to ask questions.  Moreover, the opportunity to really focus on writing code in the labs solidified the foundation that I've been trying to build.

Is Canonical using Clojure?
I don't have anything like global visibility in Canonical, but I'm pretty sure the answer is "not at this time."  The door is not closed on some experimentation, I believe, and Launchpad's intended move to a SOA could be an opening.

---

Now here are the questions I had, in no particular order, with my understanding of the answers so far.  These questions make for a pretty random collection, but maybe other people are having some of them too.

Can you deploy with Leiningen?  How are people deploying?
The simplest answer I got from that was the demo Aaron gave in training.  Most tech demos are supposed to make a task look blissfully easy, and he succeeded.  Leiningen plus a Ruby-powered Heroku gem got the app up on Heroku amazingly easily.  My notes on how he did this are non-existent, but that's because I understood that this information was going to go in the second edition of Programming Clojure, which I already have purchased as part of the beta program.  Unfortunately, looking at that chapter now, at least in beta 3, I don't actually see the deployment bits there.  I bet it will be added, though, and for now, Heroku has a nice document that's pertinent.

What if you are not using Heroku?  As I expected, I get the impression that there are many different answers.  It looks like the last conj had a presentation about continuous deployment in a commercial environment, and there are projects to help deploy to GAE, and so on.  If I had to do a commercial deployment of it, I'd have to dig into it, because I don't know.

A possible (minor?) challenge within Canonical is that the Sun JDK is preferred by Clojure AIUI. Ubuntu prefers OpenJDK (and OpenJDK is the future?  For someone?).  I already had switched my Ubuntu machine to use Sun's JDK because of my understanding that the Sun JDK was the first-class Clojure JDK.  It appears to still be true: as recently as this past week, when some trainees on Ubuntu had an issue with ClojureScript on OpenJDK they were encouraged to switch to Sun's package. [Stuart Sierra clarifies in comments below that OpenJDK works with Clojure and is part of their continuous integration tests; the only remaining problems are on the ClojureScript side.]

ClojureScript uses Google's Closure Javascript library and compiler.  Here comes Dart, which looks like it might disrupt and replace Google's Closure effort.  Do the ClojureScript devs have any connections to the Google Closure team, to have visibility on where this might go?
No. :-P

What debugging tools do Clojure developers use?  Is there an equivalent to a step debugger like gdb or pdb?
There is a step debugger, but none of the core developers I talked to were using it.

The most common answer I heard was to rip apart the composite functions and see how they are working.  That seems reasonable for debugging in the small, and has worked fine for my toy experiments.  It does not seem reasonable as a sole strategy for debugging large applications.  There are other strategies I had seen already, but the answers to this stackoverflow question to me show a bit of indecision in the community and even what I'd call "immaturity," in the sense that good standard answers still need to shake out a bit.  The impression I have after training and the conference is the same.  I'll be very curious to see where this aspect of Clojure programming goes.

I didn't ask about profiling tools, in part because I'd seen information that standard Java profilers were very effective, so that would be the obvious place to turn.

What's up with Clojure's testing attitude?  I'm getting mixed messages.  There are multiple Clojure testing frameworks, and tests integrated into the language code itself, and yet talks from prominent Clojure developers often take an ambiguous or even negative attitude towards testing, especially unit testing.
The most recent talk like this is Rich Hickey's excellent Simple Made Easy talk, which I'd encourage you to look at if you have any curiosity about Clojure's underlying ideas, and how they can be applicable to coding generally.  In it, he amusingly compares relying on tests for protecting your software from mistakes to relying on guardrails for all of your driving needs.  He throws a bit of a gauntlet out at the Agile community, but his underlying point (as he clarifies elsewhere) is not that tests are useless but that they are insufficient for writing and maintaining good programs.

However, there's still the thrown gauntlet.  You can take the gauntlet back, but why was it thrown in the first place?

Similarly, in training, Stuart Holloway made some comments about the inability of developers to know all the scenarios to put in unit tests.  I agree.  I'd say that unit tests are often compromised both by our incomplete understanding of a problem, and by a practical desire to write tests that cover "enough" of the possible scenarios.  I'd go further to say that, once you move above testing the very simplest "unit" of an application, the distinction between "unit" and "integration" tests becomes arguable; and the ability to identify and write tests to cover all possible scenarios becomes more and more unlikely.

But so then do you just not try?  That's the impression I got from that part of the presentation.  The code moved into a (very interesting) discussion and demo of a generative test framework in lieu of unit tests.

I was...surprised?  Confused?  Concerned?  Choose three.

I asked Stuart Sierra for clarification.  His perception was that, if there is a shared opinion in the Clojure community about testing, it is that automated tests are good, but insufficient; and that often people from the Agile community oversell them as a panacea.  Beyond that, Clojure developers have different opinions, as people tend to do.  Some prominent developers believe that well-written ("simple," given Rich Hickey's definition) functions should be clear enough that tests are unnecessary, and only higher-level tests of integrated behavior and assumptions are of sufficient value to merit their development and maintenance cost.  Other developers have more appreciation of lower-level unit tests.

I was happy enough with that answer.  Different strokes, and all that.

Why doesn't Clojure support reader macros?
A reader macro is a Lisp thing.  If you don't know what a reader macro is, you probably don't care, so I'm not going to try to dig in and explain it (and searching for "reader macro" didn't give me a good quick answer or link for you).

The short answer, AIUI, was that it makes reading Clojure, both for different humans and for different machines, more difficult.  That's a so-so answer, though maybe I misunderstood it.

The more interesting answer is that a number of people are interested in leveraging the Clojure reader to produce a general-purpose data exchange that is superior to JSON and XML, and to do that, they need the reader to be uniform.  Rich Hickey's conference talk had a bit more on this that I mention in the next post.

Do Clojure devs realize that the language is not easy for newbies?  Do they care?
OK, I didn't ask that precise question, and it obviously reflects an opinion.  That said, I'd say that the answer I heard during training was, "yes, and mostly".

So, here's what I found unfriendly for newbies, and the replies I heard.

• Exception formatting is rough.  I think there are some 1.3 tricks to make it better, but this problem was reiterated by others in the conference, so I don't think this is fully addressed yet.  Moreover, the trainers (I forget which one) said straight out that performance comes first: generating friendly exceptions in a way that costs performance is not acceptable in the language's core.  Reply: 1.3 already had some work in this direction.  Stuart Holloway is interested in improving this further, and it sounds like different Clojure build targets could be the hook that makes this happen: you choose whether you want pure speed or you want better exceptions when you build your Clojure.  This was discussed as a "debug" build, but it seems to me that this is a feature I might want in production.  If I can afford to have a web server in (slow) Python or Ruby, then I ought to be able to afford a web server in (much faster) Clojure, with (still much faster than Python/Ruby) nice exceptions that I can diagnose more quickly when they occur in production.
• Non-Java programmers are (currently?) second-class citizens.  Again, one of the trainers said plainly that problems for non-Java programmers are not worked on by the core developers.  Maybe that's one person's opinion, but then again, none of the other trainers threw tomatoes at him, so it probably isn't too far off.  Given that one of the main selling points of Clojure is Java interop ("Clojure") that's not unreasonable, but it's also not newbie-friendly.  Reply: I didn't raise this to anyone, so I didn't get a reply.
• I'd love to see the in-REPL documentation improved.  Examples and cross-references would be nice, and the text could be clearer.  Specifically, when I look at the docs of a function, examples would be nice.  I would like to be able to do something like "(help #"foo")" and see a list of the core functions for working with a regular expressions.  I'd like to be able to look at docs for a namespace, which might give an overview of how to use the contained functions, or at least point to a web page.  I'd like for the docstring of "for" to be more explicit about what the supported modifiers do (but look, it has an example!).  A lot of these jobs are done on the great and unofficial clojuredocs site.  It would be great to have them official and integrated.  Reply: Stuart Holloway is interested in this, and has some even better ideas along these lines, involving parsing code locally to build databases.  He mentioned this in his talk.  [I don't quite remember how one would build the database without lots of type annotations hanging around, but I think there was an expected answer.]

How do you pronounce "assoc"?

There's some variance, but watch out: you don't want to sound un-cool like me.

Yes: "uh-soash"

Yes: "uh-soass"

No: "uh-sock"

How do you pronounce "Leiningen"?

Just choose something and say it loudly.

FWIW, my best guess is "lie-ning-in".

Why does destructuring work differently in function arguments and in lets?

Because they are implemented differently.  ...Right.  In retrospect, maybe I should have dug into this more?

Example: you can say (let [[a & b :as all] a_sequence] ...code block...) and it will work, but you can't say (fn [a & b :as all] ...code block...).  You can say (fn [& [a & b :as all]] ...code block...) for the desired effect.  When I'm added to the dev list I'll bring this up again and make sure this shouldn't be filed as a bug...and maybe try to find the associated code before that.

How do you synchronize other transaction participants with the software transactional memory?

Clojure agents created within a transaction will only fire when the transaction ends successfully.  Alternatively, outside of the transaction you can create watchers on the refs you care about, and they will be fired when the ref changes at the end of a successful transaction.

Unfortunately, neither of those give you full-fledged transaction participants, so you can't have any guarantees about their interactions.  This unnecessarily limits the utility of the STM, in my opinion.  With it, you could have guarantees like "if this transaction succeeds, that means I have successfully written a record of it to a file" or "if this transaction succeeds, that means I have successfully sent a message to my message queue (with its own guarantees)".

Auto-reloading code after a change, the way many web frameworks like to do it, is difficult, fragile, and ultimately broken in Python (and, so I hear, in Ruby).  Is it better in Clojure, maybe because of the immutable data structures?

No. :-)

It happened pretty quickly in a Noir example we saw in class.

---

That's it for the questions and answers that I wrote in my notes.  Next up, talk summaries.