Notes & Learnings from Q Con London 2014 - Day 1

I was lucky enough to go to Q Con London 2014.  Here is an update of my experience from day 1.

Damian Conway - Life, The Universe and Everything

Damian is an amazing speaker and made his already fun, interesting, geeky subject, even more fun interesting, and geeky with his great presentational skills. He showed us Perl code (and later Klingonscript) that not only implemented the game of life but also (kind of) disproved Maxwell's Demon. We were also shown an example of a Turing complete machine which was implemented using game of life made by Paul Rendell.  Video also on youtube.

This talk reminded me of another cellular automaton I encountered many years ago whilst studying Genetic Algorithms at University.  I am pleased to say that the creator (David Eck) has converted the original Java Applet version of his program Eaters, that so impressed me, into javacript available here.

What I took away...Coding can and should be fun.

Daniel Schauenberg - Development, Deployment & Collaboration at Etsy

Etsy have around 150 engineers working on a monoloithic php application which forms etsy.com.  Although this sounds like an old fashioned architecture, their delivery pipeline is very impressive.  They manage 50 deploys a day.  Other points:-
  • They favour small changes.
  • 8 people can deploy different sets of changes in one build (that's there magic number).
  • The deploy process the "Deployinator" has two buttons only - no ambiguity, everything automated.
  • They use config to switch features on after the code is safely in production (i.e. soft live releases).
  • Developer VMs can run the entire stack and closely mimic production.
  • They make use of Linux Containers (LXC) On day 1 of working for etsy you deploy a change - This seems like a great idea to me.
  • Tons of dashboards which are monitored after releases.

I asked Daniel at what point etsy decide to roll a build back and stop debugging an issue, he replied that there is no rollback (the closest thing to a rollback is revert commits and create a new build).  This doesn't seem like a good idea to me. I think each build should be rollbackable.  Debugging an issue calmly after you have restored service with a rollback is surely better than a forward only stratgey.

What I took away...Continuous delivery can be achieved without a perfect architecture if you have the right processes and culture in place.  Etsy should be applauded.

Uwe Friedrichsen - Fault Tolerance & Recovery

Uwe explained a lot of common fault tolerance patterns, including...
  • Timeouts.
  • Circuit breaker.
  • Shedding load.
Uwe also explained the Netflix library Hystrix.  

What I took away... There are a lot of good patterns out there for fault tolerance, a lot of them are discussed in Release It!

Graham Steel - How I learned to stop worrying and trust crypto again

Graham started his talk by saying that "the paranoids were right" - NSA has been intervening with crypto standards.  However, all is not lost as properly implemented crypto systems can be relied on.

Here's what makes a good crypto API:-


  • Open and subject to review - More likely to have bug fixes and patches applied.
  • Supports modern cryptographic primitives.
  • Good key management - This can be the Achilles heel of a crypto API.
  • Mistake resistant - Some think people should be free to make mistakes.  I don't!
  • Interoperable.
Examples of crypto APIs.... My notes are sparse here...

  • Big success, widely used.
  • Old - Current version 2.4 was released in 2004.
  • Implementation is closed, but the standard is open.
  • The API is closed (under Oracle's control), however some providers, like Bouncy Castle are open source.
  • Has around fifty "Fix me" comments in it!
  • Contains the NSA backdoor random number generator which you are free to use or not.
  • Work in progress to provide encryption in clientside javascript - Lots of challenges here!
Lessons
  • If you roll your own cryptography implementation, you are guaranteed to have a floor in it.  
  • TLS has been around for years and they found issues just 3 days prior to this talk!
  • Applied Cryptography is a dangerous book as it encourages you to roll your own cryptography.
What I took away...  A new understanding of how much I don't know about security and cryptography.

Dave Farley - Continuous Delivery

According to Dave... "Our highest priority is to satisfy the customer through early and continuous deliveries of valuable software."


  • Finished means in production.
  • Break down silos, Business, Testers, Developers, Operations, all need to collaborate more to make it work.

Feedback loops - starting with smallest and quickest:
  1. Unit Test - Code.
  2. Executable - Build.
  3. Idea - Release.
Principles:
  • Keep everything in version control - I agree.  Any tool that has a UI-only is to be avoided in my opinion.
  • Automate everything.
  • Build quality in.
  • If something is painful, do it more often.  If you release every 6 months because it's painful, the lesson is release every 2 months, then month, then week etc etc.  Don't start releasing every year, it will get worse.
  • Manual testers are far better at exploratory creative testing than repetitive regression.

Accidental benefits:
  • When debugging a long standing defect, if version control and deploy are fast and reliable, you can determine which build introduced it.  Binary chop through a list of builds until you have narrowed it down,
  • Automating everything can introduce auditing for free.
What I took away... Dave Farley knows this subject inside and out and his, Continuous Delivery is highly recommended.

Paul Simmonds - Identity is the new currency

Data is going from: 
  1. Internal.
  2. De-perimeterized.
  3. External Collaborations.
  4. Secured Cloud.
Interested or working with security on the cloud... you must read:

Comments

  1. Anonymous11 March 2014 at 02:34

    Good overview. What needs to be done for us to release 50 times a day? :)

    ReplyDelete
  2. Ben S1 April 2014 at 00:46

    Yo - thought I'd finally comment.

    I totally agree with the Etsy engineer that there shouldn't really be such a thing as a rollback. After a fair few years working with a forward-only approach, having to do a rollback would seem like defeat now ;-)

    There are a few reasons for this:
    Cost vs Benefit - ensuring a rollback can happen costs development time, especially when we are talking about migrating schema and data. If we don't spend time here, the rollback can cause as many, if not more, issues than the actual deploy. If the data migration was long-running, it's going to double the time your site is down.

    50 deploys a day - if you are deploying 50 times a day (which in my opinion, is a great thing), chances are you are deploying a much smaller amount of code than somebody deploy once a week. This makes solving an issue with a deploy a much less intimidating prospect. e.g. lots of deploys and forward-only approach go hand-in-hand.

    Rollbacks - if you are having doing rollbacks often, there is obviously a more serious issue at play

    Aiming for a forward-only approach - if you aim for a forward-only approach, you solve a lot of other issues on the way, e.g. we need to get the dev machines closer to the production env so we can see these issues earlier.

    ReplyDelete
  3. Phill Barber20 April 2014 at 14:10

    Thanks for the comment! All good points well made.

    This is a really interesting topic and I think that there are many organisation/project specific factors which can sway the balance between the two approaches.

    Risk - The riskier the deployment/change is, the more attractive a roll-back becomes. Of course we all want low risk deploys, but sometimes you find yourself working with a legacy system with forgotten functionality. This point is very much linked to the confidence you have in your testing which is obviously very important.

    Impact of outage - The risk of an outage could be high but perhaps the impact is low. Perhaps your application/service fails in a graceful manor with say a small component being temporarily not visible to some users. Or perhaps downstream dependencies of your app/service just fail stale. The opposite of this would be big embarrassing dead-end sorry pages to all users.

    Effort of supporting a roll-back - If the rollback is complex to create and test then forward-only obviously looks more attractive. If it's simple then why not!


    I have never worked with a forward-only approach so I suppose having a rollback on hand is a source of comfort I have grown used to. However, I'm sure given the right scenario I could be persuaded to the brave forward-only approach!

    Thanks

    ReplyDelete

Post a Comment

Popular posts from this blog

Lessons learned from a connection leak in production

We recently encountered a connection leak with one of our services in production.  Here's what happened, what we did and the lessons learned... Tests and Monitoring detected the issue (silently) Our automated tests (which run in production for this service) started failing soon after the incident occurred.  We didn't immediately realise since they just went red on the dashboard and we just carried on about our business.  Our monitoring detected the issue immediately and also went red but crucially didn't email us due to an environment specific config issue. The alarm is raised A tester on the team realises that the tests had been failing for some time (some time being 4 hours... eek!) and gets us to start investigating. Myself and another dev quickly looked into what was going on with our poorly  dropwizard  app.  We could see from the monitoring dashboard that the application's health check page was in error.  Viewing the page showed us that a downstream ser
Read more

How to connect your docker container to a service on the parent host

Recently I needed a docker container to access a service running on the parent host.  A lot of the examples on the internet were very close to what I needed, but not quite. First off, docker's documentation provided an example that (as far as I can see) doesn't actually work.  The following (I think) is meant to inject the src ip address of the parent host's default interface: $ alias hostip="ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print \$2 }'" $ docker run  --add-host=docker:$(hostip) --rm -it debian What it actually does is inject the ip of the default gateway on the parent host which doesn't help us. Michael Hamrah 's blog details the " gateway approach " of getting the parent's host ip from within the docker container.  This works great, although I wanted to inject the ip on container run and not have that logic within my docker image. Inject the src ip of the docker interface I think the best
Read more

Client Side vs Server Side Session

Image
We recently looked at replacing our legacy session management system with a new one.  During this analysis, we came close to choosing a client side session but eventually concluded server side was better for us.  Here's why... Client side session In this model, all session state is stored in the client in a cookie.  The benefits of this are you don't need to worry about persisting and replicating state across nodes, session validation is lightning fast since you don't need to query any data store which means it's super scalable.  The session cookie must obviously be tamper proof (to prevent people creating a session of their choice) which is achieved by signing the cookie using asymmetric cryptography. The signing of a cookie value uses the private key, the validation uses the corresponding public key.  Our idea was to try and keep the private key as private as possible by storing it in memory only.  Each node (4 shown below) would create a new private and public
Read more