An escalation policy that doesn't wake the whole team

The simplest escalation policy is "alert everyone". It never misses, and it teaches a whole team to sleep through their phones. A better policy pages one person, gives them a fair window to respond, and only then widens the circle.

Start from the failure you are protecting against

An escalation policy exists to answer one question: what happens if the person we paged does not respond? They may be asleep, in a tunnel, on a flight, or have muted the channel. The policy is the plan for that case. Everything else is routing.

Two opposite mistakes are common:

• Too wide. Everyone is paged at once. Each person assumes someone else will pick it up, and the cost of every incident is multiplied by the size of the team.
• Too narrow. One person is paged once. If that page fails, the outage runs until a customer complains.

The fix for both is the same: time. Widen the audience only as the clock runs.

Build it in tiers

A workable shape for a small team has three tiers.

1. Tier 1: the person on call. One named human, reached by the channel most likely to interrupt them. Chat is fine for a daytime warning; for a night-time outage it should be something that rings, such as a push notification, SMS or a phone call.
2. Tier 2: a backup. A second person, paged only if tier 1 has not acknowledged within the window. They are not a co-owner. They are insurance against tier 1 being unreachable.
3. Tier 3: a last resort. A team lead, a shared channel, or a ticket. This tier is rarely reached. Its job is that the alert is never only in somebody's pocket.

Three tiers is usually enough. Every extra tier adds delay and another place for the policy to be wrong.

Choose the waits deliberately

The wait between tiers is a trade between speed and noise. Too short, and you page the backup while tier 1 is still reaching for the phone. Too long, and a dead pager costs you the better part of an hour.

A reasonable starting point for a customer-facing service is 5 to 10 minutes before tier 2 and another 10 to 15 before tier 3. Work out the worst case explicitly. If detection takes 3 minutes, tier 2 waits 10 and tier 3 waits 15, then a completely unanswered page reaches the last resort after 28 minutes. Ask whether your business can live with that number. If not, shorten the waits. If it can, resist the urge to shorten them anyway.

Internal tools, batch jobs and staging environments can have far longer waits, or no escalation at all. The policy should match what an hour of downtime costs for that service.

Acknowledgement is what stops the clock

Without an acknowledgement signal, a policy cannot tell "tier 1 is handling it" from "tier 1 never saw it". It can only keep escalating on a timer, which pages the backup even when tier 1 is already working the problem. That is exactly the noise you were trying to avoid.

So the alert itself needs a way to say "I have this". The cheapest version is a link in the alert message. One tap, no login, no hunting for the incident. Once acknowledged, the escalation stops. If nobody taps it, the next tier fires. Silence should mean "nobody has seen this", never "someone probably has".

Repeat until somebody answers

A single page is easy to miss. If nothing acknowledges, repeat the page to the current tier at an interval, say every 5 minutes, before moving on. Repetition is the cheapest escalation there is: it costs no extra people.

Put a limit on it. A page repeated forever to a phone that is switched off is just noise in the log. Repeat a few times, then move to the next tier.

Keep quiet hours honest

Not every alert deserves a 3 a.m. phone call. Decide ahead of time which alerts do:

• Page at any hour: the customer-facing service is down, a payment path is failing, data is at risk.
• Hold until morning: a certificate with three weeks left, a slow-but-working endpoint, an internal dashboard.

Mixing the two on one policy is how a team ends up muting everything. Give the urgent and the non-urgent separate paths, and send the non-urgent ones to a place that is read at working hours.

Follow the rota, not a name

Hard-coding "Alex" as tier 1 works until Alex goes on holiday. A policy that points at a role, whoever is on call right now, survives people changing. That means the rota needs to be a real, current thing and not a wiki page nobody updates. Keep it small: one person on duty at a time, no overlaps, and a clear fallback when the slot is empty. Test the fallback, because it is the one that runs on the worst night.

One policy per service, not one for the company

A global policy is simple to explain and wrong for most services. The checkout API and the internal reporting job should not share a ruleset. Let each monitor carry its own policy, and start new monitors from a copy of a sensible default rather than from nothing.

How this looks in Hesklo

In Hesklo each monitor has its own canvas, and the escalation policy is what you draw on it: a down port connected to notify steps, with waits, branches and repeats between them. Email steps can follow an on-call rota, and an acknowledge link in the email stops the escalation. The escalation policies page shows the building blocks. None of the design above depends on it, though; the same three tiers work in any tool.

A checklist to review your policy

1. Can you name tier 1, tier 2 and tier 3 for every customer-facing service right now?
2. Is the worst-case time from detection to last resort written down, and acceptable?
3. Does acknowledging an alert actually stop the escalation?
4. Is repetition capped?
5. Do urgent and non-urgent alerts take different paths?
6. Does the policy follow a rota rather than a hard-coded name?
7. Have you triggered a test alert end to end in the last quarter, including the tier 2 hop?

That last question matters most. An escalation policy you have never run is a hypothesis. Run it on a quiet afternoon, with a teammate pretending to be asleep, and fix whatever turns out to be wrong. Related reading: controlling alert noise keeps tier 1 from being paged for things that were never outages.