We had eleven dashboards. On-call engineers looked at one of them. That ratio is not a tooling problem, it is a design problem, and it took us far too long to treat it as one.

Start from the question

When a page fires at three in the morning, the engineer asks one thing before anything else: is this my problem right now, or can it wait until the morning. Every dashboard we had was organised around systems. None were organised around that question.

So we rebuilt the default view around three states and nothing else. Degraded for someone, degraded for everyone, or recovering. Everything more granular sits one click away.

Killing the noise

We deleted sixty percent of our alerts in a single afternoon. The rule we used was simple: if an alert had fired more than five times in ninety days without a human taking action, it was not an alert, it was a log line.

Six months on

Mean time to acknowledge dropped by eleven minutes. More importantly, the number of pages that woke someone up unnecessarily fell by roughly two thirds, and three engineers told us on-call stopped feeling like a tax.