Elixir: Erlang records and the Erlsom XML library

I wanted to jot down some notes about Elixir, because I'm learning it, and because some of the pieces I assembled for my most recent code exercise were hard to find across the web. Hopefully it will help someone else, or at least be something I can refer to again in the future.

I was playing around implementing the last exercise from Chapter 13 of Dave Thomas' Programming Elixir book: get the XML for the current observed weather at an airport from the NOAA's observations (example), and render it. First I used SweetXML, which is a wrapper around the standard Erlang XML library xmerl. This was pleasant and easy (once I figured out the XPath syntax for getting a tag name).


(Yes, I could have used the Erlang xmerl library directly, but it was easier for me with SweetXML.)

However, when digging around for Elixir and Erlang XML tools, I encountered the Erlang SAX-based XML library erlsom, and in particular it's "Data Binder" mode. On the face of it, it seemed like what I wanted: parse a doc into a simple data structure that I could then use elsewhere. Maybe it would be simpler than the SweetXML approach, where I had to rebuild a map from the XPath results!

This is not the case, at least in Elixir. :-) However, digging down into that rabbit hole revealed some interesting bits that were hard-ish to find, and maybe worth sharing for others. If you are interested in exploring the erlsom library from Elixir, or if you are interfacing with Erlang from Elixir and need to use Records, this might be mildly valuable for you.

To start off with the pretty bit, eventually my function to parse the XML and give me the data structure I had chosen ended up looking nice enough.


To get there took a bit of doing, though. First, you need to get the associated xsd file. Then you need to convert it to an Erlang .hrl (header or C-style include) file using an erlsom function. You can see my probably-not-particularly-fantastic way of doing that over here, if you are interested.

Then, when I needed to use that .hrl file so I could use it to interpret the scanned data from erlsom, I had to include it in Elixir somehow. I had to piece together that I was supposed to use a Record, and I found some examples on how to feed a .hrl file into it.


Other examples you'll find around use `from_lib:` instead of `from:`. I think `from_lib:` is for using data from compiled libraries, and `from:` is for files local to your project? I'm not clear, but I needed `from:`.

We'll use those in the `erlsom_transform` function you saw in the first gist. You might have also noticed the `@xsdModel` in that code. It came from this.


Now the only piece left is the `erlsom_transform` function. I wanted a map, not a Record. I needed to pattern match against a Record, which I overlooked in the docs–lines 1 and 3 below are simple examples, but you can also pattern-match on fields in the Record.


For reference, this all resulted in a data structure that looked like this. I never bothered to figure out something pleasant to do with the `anyAttribs` bits.


If I were actually interested in working with XML I might explore how to use the erlsom SAX parser to directly produce Elixir structs instead. I am just exploring though, and the world around me uses JSON, not XML.

I hope that helps someone else! If you want to see this live in my poorly tested, poorly commented exercise, have at it.

Yellow Squad Weekly Topics: December 14

Summary:

• Makyo: the GUI can break the charm
• bcsaller: Write user stories as functional tests: improv plus Selenium
• bcsaller: card velocity improving
• gary_poster: run the tests before landing, or, we are not as good as an automatic tester
• gary_poster: teknico is "documentation daddy"
• bcsaller and Makyo: proliferation of ENV=1 make [target] rules
• goodspud: what do we do with the juju-gui charm in the gui?

Yellow Squad Weekly Project Report: December 14

We are working on the Juju GUI.

This week

• Very good interaction with design team
• Improved team velocity after deceleration last week
• Real users are starting to trickle in

We have had some very positive interactions with the design team. We have dug up some good data, requirements, and decisions together, and we've increased our dialog. They have been very responsive to what we need in the short term, and are thinking creatively and practically about the long term future directions.

Our velocity has picked back up somewhat. We feel it, and an analysis of kanban board data shows it. In November, we had an average cycle time--the time from when we start a card to the time we get it landed in the trunk, including reviews and review responses--of about three days. Last week, we popped up to five, and then six days. This week, we started decreasing again, down to a five-day cycle time. Five days is still far from ideal, and arguably affected in part by some quick-fix branches that we got through the board, but we also see signs of improvement individually. Hopefully this represents a continuing trend to get us back down to a three-day cycle time.

The week ended nicely with a couple of GUI charm users getting support on IRC.

New project: Juju GUI

Welcome back! We've been away working in secret for awhile. Our work is open now, so our retrospective notes are too.

Our current project is a web-based GUI for Juju. Juju lets you deploy connected services to the cloud in a convenient, vendor-neutral, and powerful way. The GUI lets you visualize and manage your work (see also another blog; a demo of our trunk, which is reset every 15 minutes; and the project itself). We've been working together with the Cloud Ecosystem Development team at Canonical on the project, with Kapil Thangavelu as the technical architect.

At the recent Ubuntu Developer Summit, we got feedback, had user tests, sprinted, and made plans. Right now, we are working to make it easy for anyone using Juju to deploy the GUI. We also have a refactoring of the environment view in progress.

We've had a lot of project meetings before the release. I published those notes internally. Since we open-sourced, I've been swamped. We've had a number of meetings, and I have a pretty big backlog of decisions to record.

Therefore, to get back in the swing of things, the rest of this post will be a record of decisions and hints from the the past few meetings. That will catch us up, and we can proceed from there at the next meeting with notes more like what I've given in the past--both project status notes and decisions and hints, with the decisions hopefully better grouped so you can find what interests you more easily.

Yellow Squad Weekly Topics: August 17

Summary:

• bac/gary_poster: re-reconsidering required pre-implementation calls
• gary_poster: postpone reviewing the collaboration feedback experiment?

Yellow Squad Weekly Project Report: August 17

Summary:

• We're still hiring for openings in a very cool Python & cloud position, so please apply soon if you are interested!
• frankban landed a gigantic lpsetup branch to simplify code flow per discussions we've had lately in the squad (see last week's "topics" discussion about OO vs. functions, etc.).
• benji and gary_poster interviewed a lot.
• bac identified and worked on final changes before we ask for user acceptance for lpsetup. In particular, we are trying to get rid of the finish-init-target command and we are trying to clean up the top-level help text.
• benji may have fixed bug 1030148 (and now needs to integrate the fix into Launchpad).
• gary_poster managed to run parallel tests a few times. The basics are still working. The three pre-existing bugs are still there. We now also have subunit bug 1038250 and probable test interaction bug 1038253.

Yellow Squad Weekly Topics: August 10

Summary:

• gary_poster: reconciling staff unavailability with kanban work-in-progress (WIP) limits
• gary_poster: OO versus functions, inversion of control, and how much agreement is important
• gary_poster: More incremental success with our integration tests

Yellow Squad Weekly Project Report: August 10

Summary:

• A vacation kept one weekly report from being written, and another weekly meeting from happening. Welcome back!
• We're still hiring for openings in a very cool Python & cloud position, so please apply soon if you are interested! We are progressing pretty far along the process, though we have many more interviews to make. Interviews consumed gary_poster completely, and affected all of us to one degree or another.
• Interviews have kept gary_poster from running parallel tests, but we expect that our three old bug friends are still waiting for us.
• benji landed some diagnostic information that might help us figure out 974617/1011847.
• bac and frankban provided IS with detailed, tested instructions for our proposed approach for IS to deploy the LXC test setup.
• IS has approved our proposed approach to deploying the LXC Launchpad test set up. We think. We are waiting on this to be deployed.
• The lpsetup developer story was planned (largely thanks to frankban) and implemented.
• bac configured the integration tests for lpsetup to run with the unit tests prior to every commit to the main branch. We are using Juju to drive lxc-within-lxc on a Canonistack Tarmac instance. We think this is extremely cool for many reasons--not least of which are the presence of the working, gatekeeper integration tests.
• We've been given a hard date to switch to our next project: September 3. Hopefully the buildbot slaves will be deployed and debugged sufficiently by then. It's a shame we didn't get to the parallel-ec2test-via-juju plan we proposed for another stretch goal, but maybe someone else will get it approved and started.

Yellow Squad Weekly Tricks: July 20

Summary:

• bac: Don't use seteuid with bzr
• frankban: Using deb package recipes with Distribute-based Python packages
• frankban: os.path.expanduser uses $HOME to expand '~'
• frankban: In tests, use a temporary home plus a bzr whoami before bzr commit

Yellow Squad Weekly Project Report: July 20

Summary:

• This was gmb's last week with us. Farewell from Yellow squad, and best wishes for leading Green squad!
• We're still hiring for five openings in a very cool Python & cloud position, so please apply soon! The interview process largely consumed 3/5 of the squad; next week it will affect 3/4 of us. We've had some good interviewees, and we hope to have more.
• We did not have as many test runs this week because of the interviews. Our successful test run from last week stopped this week after 42 consecutive passing runs, after which we had instances of 974617/1011847 and 1014916.
• The parallel testing machines in the data center are reportedly ready for configuration and we are now waiting for IS to get to the task.
• We tested a new package of the lpsetup code, fixed some problems, and have a working package for both the developer use case and the testing use case. The next step is to design the interactive story.
• The lpsetup code has a second integration test, this time for the LXC approach.

Yellow Squad Weekly Topics: July 13

Summary:

• benji: Juju-based integration tests are a success for incremental value
• bac/gary_poster: Is it ever OK to knowingly check in broken code?
• gary_poster: Not meeting our weekly goals

Yellow Squad Weekly Tricks: July 13

Summary:

• benji/gary_poster: Different ways to keep Juju from forcing you to repeatedly accept SSH keys

Yellow Squad Weekly Project Report: July 13

Summary:

• Parallel tests are running very well this week, with 37 successful runs in a row so far...even though we haven't changed anything.
• The parallel testing machines in the data center will reportedly be ready for us next week.
• The lpsetup code's refactoring is again at a usable milestone, and we are ready to test a new package of the code.
• The lpsetup code has its first integration test, thanks to Juju.
• In the upcoming week we intend to add more integration tests and more developer-friendly features for lpsetup.
• gmb will be leaving us to start his own squad on our team!  In related news, we're hiring.

Yellow Squad Weekly Topics: July 6

Summary:

• gary_poster/benji: Personal feedback loops
• bac/benji/gary_poster: Poorly specified tasks

Intentional duplication of effort: learning and prototyping

When do you want to duplicate effort?  You often want to duplicate learning across a team.  What about prototyping?

Yellow Squad Weekly Tricks: July 6

Summary:

• bac: manage your services together with their supporting infrastructure
• gmb: beware: Launchpad's ec2 command will sometimes silently break if your download cache doesn't have what you need.
• gary_poster/gmb: if you get a weird ec2 result, don't forget about ec2's postmortem option.

Yellow Squad Weekly Project Report: July 6

Summary: We made some good progress, but this week was not as successful as we had hoped.

Yellow Squad Weekly Retrospective Minutes: June 29

Introduction

What is this post?

I'm the lead for the "Yellow" squad in Canonical's collection of geographically distributed, agile squads.  We're directed to work as needed on various web and cloud projects and technologies.  Every Friday, our squad has a call to review what happened in the past week and see what we can learn from it.  We follow a simple, evolving format that we keep track of on a wiki.  This post contains the minutes of one of those meetings.

Yellow Squad Weekly Retrospective Minutes: June 22

Introduction

What is this post?

I'm the lead for the "Yellow" squad in Canonical's collection of geographically distributed, agile squads.  We're directed to work as needed on various web and cloud projects and technologies.  Every Friday, our squad has a call to review what happened in the past week and see what we can learn from it.  We follow a simple, evolving format that we keep track of on a wiki.  This post contains the minutes of one of those meetings.

Our checklist for running a daily meeting with kanban

The squad I manage is fully distributed, like most of Canonical's employees.  We are all in relatively overlapping timezones--ranging from Italy to the American east coast--but none of us work in the same building, or even the same city.

We also are on the Lean journey.  We're not nearly as far along as others you'll find blogging on the web.  However, maybe we're interesting because of the combination: lean, plus a distributed workforce.

I suspect that working in a distributed team increases the importance of good practices.  When they are not followed, the pain is more intense.  For example, I think that writing everything down is important for running a business well; but it is essential, or more obviously essential, for a fully distributed team.

Another good practice like this is a daily meeting.  We've done this for many years now, and it is essential to our coordination.  We combine it with an online kanban board, which is central to our daily workflow and coordination.

We've also recently combined it with another good idea, checklists, and that's what I want to share right now.