Change the characteristics of your complexity

I was lucky enough to attend a talk by John Turner from Paddy Power as part of London Continuous Delivery meetup.  The talk was very informative and gave the audience some great insight into rolling out PaaS and continuous delivery in the real world.

In order to innovate faster, Paddy Power made a number of changes to "re-orient" the company and put the engineer team first.  To help speed up engineering, where possible, decisions on architecture were removed and replaced with convention.

Pipeline

An abstract (no tools specified) pipeline was defined with the following stages:

1. Code commit

Trunk based commits with feature switches.

2. Build

Component and Integration tests
Code quality rules (enforced by sonar)

3. Acceptance Tests

Run against the application/service as delpoyed in a VM as provisioned by Gigaspace's Cloudify product.  Config for the VM's dependencies (e.g. java installation) is defined in a blueprint.  Acceptance tests in BDD style with JBehave and Thucydides.

4. Performance Test

The goal here is to see if there has been a significant degradation in performance.  The goal is not to try and measure absolute performance.  This is performed by JMeter.

5. Stability Test

At this stage, the acceptance are run with some deliberate random failures added to various dependencies.  This has been achieved by an in-house implementation of Netflix's SimianArmy.

6. Manual Test

This is some times required for new features, legal issues and also as John put it "that warm fuzzy feeling"

7. Production deploy


No Rollbacks

I have debated about this a few times with various people (see here for my thoughts) and got into the same debate with John after his talk.  John had a great analogy... rollbacks are like car airbags, they encourage drivers to take risks.  On the other hand, a spike on the steering wheel is your roll-forward only approach and encourages a lot more care :)

Other snippets

"You can't remove complexity from an inherently complex system.  You can however change the characteristics of the complexity, by distributing it" - This is the goal of moving to a microservices architecture.

Getting your organisation to change can be hard.  A technique John mentioned was to trial out the change on a small scale, prove it works, and then to try and convince others... sound advice!

Comments

Post a Comment

Popular posts from this blog

Lessons learned from a connection leak in production

We recently encountered a connection leak with one of our services in production.  Here's what happened, what we did and the lessons learned... Tests and Monitoring detected the issue (silently) Our automated tests (which run in production for this service) started failing soon after the incident occurred.  We didn't immediately realise since they just went red on the dashboard and we just carried on about our business.  Our monitoring detected the issue immediately and also went red but crucially didn't email us due to an environment specific config issue. The alarm is raised A tester on the team realises that the tests had been failing for some time (some time being 4 hours... eek!) and gets us to start investigating. Myself and another dev quickly looked into what was going on with our poorly  dropwizard  app.  We could see from the monitoring dashboard that the application's health check page was in error.  Viewing the page showed us tha...
Read more

How to connect your docker container to a service on the parent host

Recently I needed a docker container to access a service running on the parent host.  A lot of the examples on the internet were very close to what I needed, but not quite. First off, docker's documentation provided an example that (as far as I can see) doesn't actually work.  The following (I think) is meant to inject the src ip address of the parent host's default interface: $ alias hostip="ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print \$2 }'" $ docker run  --add-host=docker:$(hostip) --rm -it debian What it actually does is inject the ip of the default gateway on the parent host which doesn't help us. Michael Hamrah 's blog details the " gateway approach " of getting the parent's host ip from within the docker container.  This works great, although I wanted to inject the ip on container run and not have that logic within my docker image. Inject the src ip of the docker interface I think the best ...
Read more

Client Side vs Server Side Session

Image
We recently looked at replacing our legacy session management system with a new one.  During this analysis, we came close to choosing a client side session but eventually concluded server side was better for us.  Here's why... Client side session In this model, all session state is stored in the client in a cookie.  The benefits of this are you don't need to worry about persisting and replicating state across nodes, session validation is lightning fast since you don't need to query any data store which means it's super scalable.  The session cookie must obviously be tamper proof (to prevent people creating a session of their choice) which is achieved by signing the cookie using asymmetric cryptography. The signing of a cookie value uses the private key, the validation uses the corresponding public key.  Our idea was to try and keep the private key as private as possible by storing it in memory only.  Each node (4 shown below) would create a new priva...
Read more