Thoughts from Sam Newman's - Microservices Data Decomposition Talk

 I was fortunate to attend a great talk by Sam Newman on the above subject, here is a brief summary of the content and my thoughts.

First off, I should say that I went in to the talk regarding myself as "experienced" on the subject of microservices.  I anticipated learning about the more complex "data decomposition" aspects rather than the basics.  However I have to admit that even the fundamental points on microservice were still a benefit for me to hear.  This is likely due to Sam being such a good teacher.  I found myself somewhat re-learning topics which I hadn't anticipated.

Microservices and Backwards Compatability

Sam described microservices as an architecture style which allow you to partition your functionality into separate services which are independently deployable.  The data that each service stores is hidden within the microservice boundary.  They allow you to scale your organisation since you can have teams working on separate systems in parallel.  

Tolerant Reader Pattern

Backwards compatible releases are achieved with the "tolerant reader pattern" (i.e. clients not failing when they encounter new fields in a response payload).  Some tools are bad for introducing this such as Java's RMI and any libraries which autogenerates code from a schema.  Martin Fowler talks about this here too.

Services shouldn't over expose

Whilst clients shouldn't read every single field, the services should only expose the minimum that is required.  The best way to do this is to speak to the actual clients.  Perhaps they only need to know the total number of Apples in the inventory service and not the row and shelf in the warehouse that they are located.  I have seen this problem first hand too!

Shared Domain Model should be avoided

I have seen this problem first hand and the downsides.  Dave Farley's great 15 minute video also explains the same point:  Code duplication is fine if it avoids coupling which is far worse a problem.  To quote Sam: 

"My general rule of thumb: don’t violate DRY within a microservice, but be relaxed about violating DRY across all services. The evils of too much coupling between services are far worse than the problems caused by code duplication."

JSON isn't the best for microservices

As soon as this was said, people started asking for further info.  Sam's point is that:
  • It's human readable (but maybe you don't care if this is machine to machine)
    • I'd argue that even if your service is purely machine to machine - human readable is a massive benefit for debugging
  • Not compact
    • Apparently it's about the same size as an xml representation - this surprised me.
Proto Buff is a good alternative as it can be faster to serialize and deserialize given it is binary if you are willing to sacrifice the human readability.

Management Information / Analytical Data

Multiple people (including myself) asked "how do you query/join data from multiple sources when its in separate services' data stores" .  I am not clear on the recommendation here.  Sam mentioned that there are two main options; Data warehouse or Data Mesh.  I think I need to read up on this.  This looks a good place to start: https://martinfowler.com/articles/data-mesh-principles.html 

Static Reference Data - Where to put it?

Country codes are a great example of this, you might have multiple services which all need this data - so where does it go?  There are multiple options and the answer will depend on the impact of your services being out of sync.

  • Separate service - perhaps this is overkill but it guarantees a single view of the data.  Clients can cache the data to speed things up.
  • Copy Paste / Bundle it in every service.  This can be achieved cleanly with a shared library.  This might be good enough if you know the impact of the data being inconsistent.  But make sure you are not creating services which have to be released all together.

The horror!

In terms of delivery, Sam said:

"You won't appreciate the true horror, pain and suffering of microservices until you're running them in production"

This was used to point out that microservices should be released little and often.  I guess the smaller the system you have in production the less the horror!

Further Reading

Comments

Post a Comment

Popular posts from this blog

Lessons learned from a connection leak in production

We recently encountered a connection leak with one of our services in production.  Here's what happened, what we did and the lessons learned... Tests and Monitoring detected the issue (silently) Our automated tests (which run in production for this service) started failing soon after the incident occurred.  We didn't immediately realise since they just went red on the dashboard and we just carried on about our business.  Our monitoring detected the issue immediately and also went red but crucially didn't email us due to an environment specific config issue. The alarm is raised A tester on the team realises that the tests had been failing for some time (some time being 4 hours... eek!) and gets us to start investigating. Myself and another dev quickly looked into what was going on with our poorly  dropwizard  app.  We could see from the monitoring dashboard that the application's health check page was in error.  Viewing the page showed us tha...
Read more

How to connect your docker container to a service on the parent host

Recently I needed a docker container to access a service running on the parent host.  A lot of the examples on the internet were very close to what I needed, but not quite. First off, docker's documentation provided an example that (as far as I can see) doesn't actually work.  The following (I think) is meant to inject the src ip address of the parent host's default interface: $ alias hostip="ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print \$2 }'" $ docker run  --add-host=docker:$(hostip) --rm -it debian What it actually does is inject the ip of the default gateway on the parent host which doesn't help us. Michael Hamrah 's blog details the " gateway approach " of getting the parent's host ip from within the docker container.  This works great, although I wanted to inject the ip on container run and not have that logic within my docker image. Inject the src ip of the docker interface I think the best ...
Read more

Client Side vs Server Side Session

Image
We recently looked at replacing our legacy session management system with a new one.  During this analysis, we came close to choosing a client side session but eventually concluded server side was better for us.  Here's why... Client side session In this model, all session state is stored in the client in a cookie.  The benefits of this are you don't need to worry about persisting and replicating state across nodes, session validation is lightning fast since you don't need to query any data store which means it's super scalable.  The session cookie must obviously be tamper proof (to prevent people creating a session of their choice) which is achieved by signing the cookie using asymmetric cryptography. The signing of a cookie value uses the private key, the validation uses the corresponding public key.  Our idea was to try and keep the private key as private as possible by storing it in memory only.  Each node (4 shown below) would create a new priva...
Read more