Alerts should treat your Ops team like the police... only contact them in real emergencies!

Every time an alert notifies your Ops team, there should be a real problem.  If there isn't a real problem, you're wasting their time, adding confusion and making it harder for them to respond to real incidents.

The problem we have: Too many alerts for non-issues and non-critical issues.

I sat with a member of our Ops team recently and was horrified to see how many notifications they received for our monitoring systems.  At times, it was almost impossible to make any sense from them due to the sheer volume of emails filling their inbox.  

Why is this such a problem?

Multiple reasons:

  • Each notification comes at a cost of lessening the impact of all other notifications.  Take this to the extreme, where Ops receive hundreds per day, the impact of an alert can be almost zero.
  • A false alarm is a distraction and can waste valuable minutes in debugging real issues.


How did we get here?

The short answer is: by diligently adding more alerts but not diligently reviewing their behaviour.  We (myself included) are guilty of the crime... Wasting Ops Time!  


Addressing the problem

This is not an easy problem to solve and will take time and effort to fix.  

Our Ops team is now (rightfully) putting pressure on the development teams to clear up the worst offending alerts.  They routinely send emails of the top ten "flapping" alerts (i.e. constantly cycling between problem and recovery) in an effort to reduce the spam.  Exposing the problem is definitely the first step towards solving it.

Review a weeks worth of your notifications... for each one ask yourself:

  • Does this alert reflect a real issue with our service that needs to be fixed?
  • Was this an issue that Ops really needed to know about?
  • What is the worst case scenario to end users given the alert?
  • Could the alert be safely set to only warn? (e.g. never notify, and only display a yellow/amber as opposed to red on your dashboard).

Some alerts will be a lot easier to fix than others, but you can easily prioritise by the number of notifications sent in a given period.  

Final point: Get included on your systems alerts (if not already), remove your email filters so they come straight to your inbox and feel Ops' pain.  Filters offer an all too convenient way of burying your head in the sand. 

Comments

Post a Comment

Popular posts from this blog

Lessons learned from a connection leak in production

We recently encountered a connection leak with one of our services in production.  Here's what happened, what we did and the lessons learned... Tests and Monitoring detected the issue (silently) Our automated tests (which run in production for this service) started failing soon after the incident occurred.  We didn't immediately realise since they just went red on the dashboard and we just carried on about our business.  Our monitoring detected the issue immediately and also went red but crucially didn't email us due to an environment specific config issue. The alarm is raised A tester on the team realises that the tests had been failing for some time (some time being 4 hours... eek!) and gets us to start investigating. Myself and another dev quickly looked into what was going on with our poorly  dropwizard  app.  We could see from the monitoring dashboard that the application's health check page was in error.  Viewing the page showed us tha...
Read more

How to connect your docker container to a service on the parent host

Recently I needed a docker container to access a service running on the parent host.  A lot of the examples on the internet were very close to what I needed, but not quite. First off, docker's documentation provided an example that (as far as I can see) doesn't actually work.  The following (I think) is meant to inject the src ip address of the parent host's default interface: $ alias hostip="ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print \$2 }'" $ docker run  --add-host=docker:$(hostip) --rm -it debian What it actually does is inject the ip of the default gateway on the parent host which doesn't help us. Michael Hamrah 's blog details the " gateway approach " of getting the parent's host ip from within the docker container.  This works great, although I wanted to inject the ip on container run and not have that logic within my docker image. Inject the src ip of the docker interface I think the best ...
Read more

Client Side vs Server Side Session

Image
We recently looked at replacing our legacy session management system with a new one.  During this analysis, we came close to choosing a client side session but eventually concluded server side was better for us.  Here's why... Client side session In this model, all session state is stored in the client in a cookie.  The benefits of this are you don't need to worry about persisting and replicating state across nodes, session validation is lightning fast since you don't need to query any data store which means it's super scalable.  The session cookie must obviously be tamper proof (to prevent people creating a session of their choice) which is achieved by signing the cookie using asymmetric cryptography. The signing of a cookie value uses the private key, the validation uses the corresponding public key.  Our idea was to try and keep the private key as private as possible by storing it in memory only.  Each node (4 shown below) would create a new priva...
Read more