Phill Barber's Blog

Recently, me and a team hit a big limitation with Ansible that really took me by surprise.   For loops, can only iterate over a single task and not roles or playbooks .  This post will discuss why it was a surprise, what we did to get around it and (briefly) what Ansible 2 introduces. Pre Ansible 2 The original issue really took me by surprise since the  documentation on for loops showed a huge number of possibilities (including conditionals, looping over arrays etc etc) that suggested to me it was Turing-Complete and easily capable of meeting all our looping needs.  So when I needed to assign a role multiple times (dynamically from a list) I was shocked to find that it couldn't be done. I'm not the only one who was caught out by this.  See here , here and here . Why this wasn't supported really puzzled me.  Some on my team concluded that it must be due to the way Ansible works internally and would perhaps require a lot of re-writing t...

Recently I was involved with setting up a build and deploy system for a suite of microservices. Each service was bundled into its own Docker image and deployed by Ansible.  One of the biggest lessons I learnt is the subject of this post. Background Project encompasses around twenty microservices (this figure only set to increase). All packaged and deployed as Docker images. Deployment by Ansible.    Some services need to be deployed to different machines on different networks since some are public facing, some backend, some internal etc etc. Different environment variables need to be injected into different running Docker containers. Each service lives in its own git repo. Each service built by CI server which runs project tests and creates a Docker image.  Big effort to minimise the amount of work/config required to create a new microservice. The Problem Where to store the ansible depl...

Here are my notes and thoughts on the Ansible meetup hosted by Equal Experts on 16th July 2015.  I benefited hugely from the event since I got to here about a  technology that I thought I knew reasonably well being used in a completely different way than I had ever considered.  Talk 1: Debugging custom Ansible modules - Alex Leonhardt   Unfortunately I missed this as I left work late : His slides can be found here: http://www.slideshare.net/aleonhardt/debugging-ansible-modules Talk 2: AWS, CloudFormation, CodeDeploy, and some Ansible fun - Oskar Pearson Oskar Pearson talked to us about how the deployment of the E-Petition's website works.  The entire deployment config has been open sourced on github here so you can work out exactly how it has been done.  Oskar's full presentation can be found here but a very brief summary is shown below so that my thoughts have some background context. Problem Domain: Busy Site. No permanent Ops team. ...

A reverse proxy such as NGINX or Apache seems to be a common feature of many systems, yet they often lack the tests associated with the business logic in the application(s) it proxies.  Here I will show how, with the benefit of Docker and WireMock , to write tests that verify your config in isolation. Git project here . The Problem   Without tests for a reverse proxy, bugs are found on deployment in a real environment with some other unrelated suite of acceptance tests failing.  This has many drawbacks: Bugs are found later (typically after committing, building and deploying) which increases your edit, compile, debug cycle. It's impossible to approach your development in a TDD fashion. Without tests in the same code base, you have to rely on comments or documentation to describe the intended behaviour. The Solution    Test your proxy in isolation, end to end, using WireMock and Docker . Docker allows us to build and run the proxy in a ...

Ansible roles can be difficult to write tests for.  They change the state of a machine and may not be able to run on the distribution of linux that your development or CI machine runs.   Docker solves both of these problems very nicely. Here I will discuss how with an example project on github . Background Before adopting this strategy I found myself testing ansible roles on the actual machine they were being deployed to.  This is bad because: It's error prone (previous failed attempts can leave the machine your testing on in a state that is invalid). You have to write more code to rollback what ansible has done so you can test again. Not being able to run locally increases you edit, compile, debug cycle. In order to automate this for CI you need to set aside a machine/VM. How to test an ansible role Jeff Geerling has a great blog post on testing ansible roles with travis ci which was adapted from his book Ansible for DevOps .  His post shows you how ...

Choosing a suitable monitoring framework for your system is important.  If you get it wrong you might find yourself having to re-write your checks and setup something different (most likely) at great cost.  Recently I looked into a few monitoring frameworks for a system and came to a few conclusions which I'll share below. Background The System At the time of this investigation, the system (which has a microservices architecture) was in the process of being "productionised".  It had no monitoring in place and had never been supported in production.  The plan was to introduce monitoring so that it could be supported and monitored 24x7 with the hope of achieving minimal downtime. Warning - I am biased Before we get started, I have to acknowledge a few biases I have.  I have worked with nagios in the past and found it to be  bit of a pain.  However, this was probably due to the fact we created our checks in puppet which added an extra layer o...

Recently I needed to share an Ansible inventory with multiple playbooks in different code repos.  I couldn't find a way to do this nicely without writing some new code which I'll share here. Trisha Gee sums up how I feel about this...  "I'm doing something that no-one else seems to be doing (or talking about). That either means I'm doing something cool, or something stupid."   Perhaps I missed an ansible feature, if so, please let me know!  I am new to it.  If not, perhaps ansible could add support for this. EDIT 19/7/2015 I'm now of the opinion that this is a bad idea.  Regarding the Trisha Gee quote above - I now know what it was I was doing (hint... not cool).   I am no longer sharing inventories as now all of our ansible deployment config (except a few roles) are in one git repo.  This same repo holds our inventories so no need for sharing.  If I did have to share them, I'd use git sub modules.  The solution detailed ...

Recently I needed a docker container to access a service running on the parent host.  A lot of the examples on the internet were very close to what I needed, but not quite. First off, docker's documentation provided an example that (as far as I can see) doesn't actually work.  The following (I think) is meant to inject the src ip address of the parent host's default interface: $ alias hostip="ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print \$2 }'" $ docker run  --add-host=docker:$(hostip) --rm -it debian What it actually does is inject the ip of the default gateway on the parent host which doesn't help us. Michael Hamrah 's blog details the " gateway approach " of getting the parent's host ip from within the docker container.  This works great, although I wanted to inject the ip on container run and not have that logic within my docker image. Inject the src ip of the docker interface I think the best ...

Connection leaks are easy to introduce, hard to spot, can go undetected for a long time and have disastrous consequences. It's not enough to fix a connection leak or code against them, you must write tests that would fail in their presence.  Here I will discuss a number of techniques that could apply to many different languages and frameworks.  I have also provided some concrete examples on github which make use of Dropwizard , JerseyClient and WireMock . What is a connection leak? Each time system A talks to system B over a protocol like http, it requires a TCP/IP connection to send and receive data.  Connections can be re-used for different requests, via a connection pool, or a new connection can be established for each request.  Once system A has finished with the connection, it needs to either return the connection to the pool, or close it.  If this doesn't happen you have a connection leak.   Given a connection leak, everythin...