Yellow Squad Weekly Tricks: July 6

Summary:

• bac: manage your services together with their supporting infrastructure
• gmb: beware: Launchpad's ec2 command will sometimes silently break if your download cache doesn't have what you need.
• gary_poster/gmb: if you get a weird ec2 result, don't forget about ec2's postmortem option.

[introduction] [project report] [tricks] [topics]

bac: manage your services together with their supporting infrastructure

We have a service running persistently on our company's private cloud (the service is tarmac, as discussed in past weeks). We occasionally need to restart this instance for various reasons. Rather than trying to communicate the new IP for every new instance among our team manually, bac had the great idea to use dynamic DNS to make a persistent reference to the ephemeral machine.

The ddclient package can automate this nicely. Brad's first inclination was to run code this locally on his own persistent, non-ephemeral server.

However, he soon realized that he could run the software instead on the cloud – on the same machine that was running our persistent service. This made configuration simpler and, arguably more importantly, kept him from having to provide some kind of reliability guarantee to the company for his own personal machines.

The one challenge is how to store passwords and other authentication tokens in a way that is secure, shared among those who may restart the service, and also friendly to automation.  We solve it with private branches, protected with SSH, managed by launchpad.

The overarching lesson is to keep your services managed together with their support infrastructure as much as possible.

gmb: beware: Launchpad's ec2 command will sometimes silently break if your download cache doesn't have what you need.

This is really specific to Launchpad's "ec2" command.  If you use that command, and you get back a failure email without explanation, and you updated the download-cache in your branch, make sure that you actually committed/pushed your changes to the right branch.

gary_poster/gmb: if you get a weird ec2 result, don't forget about ec2's postmortem option.

Again, if you use Launchpad's "ec2" command and you get seemingly inexplicable results, you might not know or remember that it has a "--postmortem" (or "-p") option that will keep you attached to the instance as it runs tests, and leave you connected after the tests run (or fail to run) so that you can figure out what happened.