Alert fatigue is a design problem: a pruning checklist
Teams do not ignore alerts because they are careless. They ignore them because the alerts are badly designed. A way to audit a month of pages and cut the noise.
When a team starts muting a channel or swiping away pages without reading them, the usual diagnosis is discipline. It is almost never discipline. Every alert that does not need a human trains the humans to expect that the next one does not either, and eventually they are right about the wrong one.
Fatigue is a property of the alerting design, so it can be fixed by changing the design. This post is a method for doing that with data you already have: last month's pages.
Start with a rule for what may page a person
Before touching any monitor, write down the test an alert must pass to interrupt someone. A workable one has three parts:
- Something is wrong now, or will be soon enough that waiting costs something. Not "something was odd a minute ago".
- A human can do something about it. If the right response is "wait and see", a person is the wrong recipient.
- It cannot wait until morning, or the person receiving it is already working.
Anything that fails the test is not necessarily useless. It is just not a page. It may belong in a chat channel, a daily summary, or a log. The audit below is mostly the work of sorting alerts into those buckets honestly.
The audit: label every page from the last month
Export or list every alert that reached a person in the last 30 days. For each one, record four things: which monitor, what time, what the responder did, and how long it took. Then give each a label.
- Actionable. A real problem, and someone did something that changed the outcome.
- Real but unactionable. The problem was real, but nobody could or needed to act: a provider outage you could only watch, or a blip that fixed itself before anyone opened a laptop.
- Planned. A deploy, reboot or migration that you caused on purpose.
- False. The service was fine. The check, the network on the sending side, or a threshold was wrong.
- Duplicate. The same incident delivered again, or to several people, or through several channels.
Be strict. "It was real, but it recovered" goes under unactionable, not actionable. The useful number is the share of pages that were actionable. If it is under half, the team's reaction to pages is a reasonable response to the data.
A fix for each label
False pages: raise the bar for "down"
These are the cheapest to remove, because the alert should never have existed. Require several failed checks in a row before a monitor counts as down, and set the timeout from the endpoint's real slow tail instead of a default. The arithmetic behind that is in why a check should fail three times before it pages. If failures cluster across many unrelated monitors at the same moment, suspect the sender, not the targets; a second vantage point is the cure there.
Planned pages: tell the system about the work
If a deploy pages someone, the deploy process is missing a step. Pause the affected monitors for the length of the work, with an automatic resume time so a forgotten pause does not become a blind spot for weeks. Pause only what the work touches. And do not use a pause as a cure for a monitor that is merely noisy; that hides the problem and the visibility with it. Maintenance windows cover the mechanics.
Unactionable pages: change the recipient or the channel
A real problem that nobody can fix still deserves to be recorded, but it does not deserve a 3 a.m. phone call. Move these to a quiet channel, or hold them to working hours. A branch of the policy for "important but not urgent" next to a branch for "wake someone" is usually enough. A schedule gate is one way to hold the first branch to the working day while the critical one stays open around the clock.
Duplicates: one incident, one message per destination
Count how many messages a single outage generated. If an incident that lasted ten minutes produced thirty notifications, the design is repeating itself. Make each step fire once per incident by default. Add a cooldown to services that flap, so each transition does not produce its own message. Repeats are fine when they are deliberate, for example a page that keeps going until somebody acknowledges it. That is noise you chose, and it ends when a person responds. The distinction between the two is covered on the alert noise control page.
Check the routing, not just the thresholds
Some of the worst fatigue comes from alerts that are correct but sent to everyone. A page that goes to the whole team teaches nine people to assume that someone else has it. Send the first alert to one person, and widen only if they do not respond. That is what an escalation policy that does not wake the whole team is for. Pair it with a rota so that "one person" is a named, current answer; see on-call scheduling.
Where Hesklo fits
In Hesklo the confirmation threshold and timeout are set per monitor, notify steps carry a cooldown and fire once per incident unless you ask otherwise, and pausing a monitor can resume itself at a time you set. The point of the audit is that you can apply every fix above with any tool that has these controls.
The checklist
- Write the page test down: wrong now, a human can act, cannot wait.
- Label last month's pages as actionable, unactionable, planned, false or duplicate.
- Compute the actionable share. Treat anything under half as a design defect.
- False: raise the failure threshold, fix the timeout, look for a shared cause on the sending side.
- Planned: pause the monitors for the work, with an auto-resume time.
- Unactionable: move to a quiet channel or hold to working hours.
- Duplicate: fire once per incident, add a cooldown for flapping services.
- Route the first page to one person, not the whole team.
- Repeat the audit in a month and compare the actionable share. If it did not improve, the changes were cosmetic.
Do not aim for zero alerts. Aim for a page that always means something, so the day it arrives at 3 a.m. it gets read.