Elixir: Erlang records and the Erlsom XML library

I wanted to jot down some notes about Elixir, because I'm learning it, and because some of the pieces I assembled for my most recent code exercise were hard to find across the web. Hopefully it will help someone else, or at least be something I can refer to again in the future.

I was playing around implementing the last exercise from Chapter 13 of Dave Thomas' Programming Elixir book: get the XML for the current observed weather at an airport from the NOAA's observations (example), and render it. First I used SweetXML, which is a wrapper around the standard Erlang XML library xmerl. This was pleasant and easy (once I figured out the XPath syntax for getting a tag name).


(Yes, I could have used the Erlang xmerl library directly, but it was easier for me with SweetXML.)

However, when digging around for Elixir and Erlang XML tools, I encountered the Erlang SAX-based XML library erlsom, and in particular it's "Data Binder" mode. On the face of it, it seemed like what I wanted: parse a doc into a simple data structure that I could then use elsewhere. Maybe it would be simpler than the SweetXML approach, where I had to rebuild a map from the XPath results!

This is not the case, at least in Elixir. :-) However, digging down into that rabbit hole revealed some interesting bits that were hard-ish to find, and maybe worth sharing for others. If you are interested in exploring the erlsom library from Elixir, or if you are interfacing with Erlang from Elixir and need to use Records, this might be mildly valuable for you.

To start off with the pretty bit, eventually my function to parse the XML and give me the data structure I had chosen ended up looking nice enough.


To get there took a bit of doing, though. First, you need to get the associated xsd file. Then you need to convert it to an Erlang .hrl (header or C-style include) file using an erlsom function. You can see my probably-not-particularly-fantastic way of doing that over here, if you are interested.

Then, when I needed to use that .hrl file so I could use it to interpret the scanned data from erlsom, I had to include it in Elixir somehow. I had to piece together that I was supposed to use a Record, and I found some examples on how to feed a .hrl file into it.


Other examples you'll find around use `from_lib:` instead of `from:`. I think `from_lib:` is for using data from compiled libraries, and `from:` is for files local to your project? I'm not clear, but I needed `from:`.

We'll use those in the `erlsom_transform` function you saw in the first gist. You might have also noticed the `@xsdModel` in that code. It came from this.


Now the only piece left is the `erlsom_transform` function. I wanted a map, not a Record. I needed to pattern match against a Record, which I overlooked in the docs–lines 1 and 3 below are simple examples, but you can also pattern-match on fields in the Record.


For reference, this all resulted in a data structure that looked like this. I never bothered to figure out something pleasant to do with the `anyAttribs` bits.


If I were actually interested in working with XML I might explore how to use the erlsom SAX parser to directly produce Elixir structs instead. I am just exploring though, and the world around me uses JSON, not XML.

I hope that helps someone else! If you want to see this live in my poorly tested, poorly commented exercise, have at it.

Yellow Squad Weekly Topics: December 14

Summary:

• Makyo: the GUI can break the charm
• bcsaller: Write user stories as functional tests: improv plus Selenium
• bcsaller: card velocity improving
• gary_poster: run the tests before landing, or, we are not as good as an automatic tester
• gary_poster: teknico is "documentation daddy"
• bcsaller and Makyo: proliferation of ENV=1 make [target] rules
• goodspud: what do we do with the juju-gui charm in the gui?

Yellow Squad Weekly Project Report: December 14

We are working on the Juju GUI.

This week

• Very good interaction with design team
• Improved team velocity after deceleration last week
• Real users are starting to trickle in

We have had some very positive interactions with the design team. We have dug up some good data, requirements, and decisions together, and we've increased our dialog. They have been very responsive to what we need in the short term, and are thinking creatively and practically about the long term future directions.

Our velocity has picked back up somewhat. We feel it, and an analysis of kanban board data shows it. In November, we had an average cycle time--the time from when we start a card to the time we get it landed in the trunk, including reviews and review responses--of about three days. Last week, we popped up to five, and then six days. This week, we started decreasing again, down to a five-day cycle time. Five days is still far from ideal, and arguably affected in part by some quick-fix branches that we got through the board, but we also see signs of improvement individually. Hopefully this represents a continuing trend to get us back down to a three-day cycle time.

The week ended nicely with a couple of GUI charm users getting support on IRC.

New project: Juju GUI

Welcome back! We've been away working in secret for awhile. Our work is open now, so our retrospective notes are too.

Our current project is a web-based GUI for Juju. Juju lets you deploy connected services to the cloud in a convenient, vendor-neutral, and powerful way. The GUI lets you visualize and manage your work (see also another blog; a demo of our trunk, which is reset every 15 minutes; and the project itself). We've been working together with the Cloud Ecosystem Development team at Canonical on the project, with Kapil Thangavelu as the technical architect.

At the recent Ubuntu Developer Summit, we got feedback, had user tests, sprinted, and made plans. Right now, we are working to make it easy for anyone using Juju to deploy the GUI. We also have a refactoring of the environment view in progress.

We've had a lot of project meetings before the release. I published those notes internally. Since we open-sourced, I've been swamped. We've had a number of meetings, and I have a pretty big backlog of decisions to record.

Therefore, to get back in the swing of things, the rest of this post will be a record of decisions and hints from the the past few meetings. That will catch us up, and we can proceed from there at the next meeting with notes more like what I've given in the past--both project status notes and decisions and hints, with the decisions hopefully better grouped so you can find what interests you more easily.

Yellow Squad Weekly Topics: August 17

Summary:

• bac/gary_poster: re-reconsidering required pre-implementation calls
• gary_poster: postpone reviewing the collaboration feedback experiment?

Yellow Squad Weekly Project Report: August 17

Summary:

• We're still hiring for openings in a very cool Python & cloud position, so please apply soon if you are interested!
• frankban landed a gigantic lpsetup branch to simplify code flow per discussions we've had lately in the squad (see last week's "topics" discussion about OO vs. functions, etc.).
• benji and gary_poster interviewed a lot.
• bac identified and worked on final changes before we ask for user acceptance for lpsetup. In particular, we are trying to get rid of the finish-init-target command and we are trying to clean up the top-level help text.
• benji may have fixed bug 1030148 (and now needs to integrate the fix into Launchpad).
• gary_poster managed to run parallel tests a few times. The basics are still working. The three pre-existing bugs are still there. We now also have subunit bug 1038250 and probable test interaction bug 1038253.

Yellow Squad Weekly Topics: August 10

Summary:

• gary_poster: reconciling staff unavailability with kanban work-in-progress (WIP) limits
• gary_poster: OO versus functions, inversion of control, and how much agreement is important
• gary_poster: More incremental success with our integration tests