Why an uptime check should fail three times before it pages

A single failed HTTP check is not an outage. It is one request that did not come back in time, and the cause may be the target, the network in between, or the machine that sent it. Paging a person on that one data point is the most common way a monitoring setup teaches a team to ignore it.

The arithmetic of a false alert

Suppose a healthy service has a 1% chance that any single check fails for reasons unrelated to the service: a dropped packet, a resolver hiccup, a slow TLS handshake. That figure is invented for illustration, but the shape of the result does not depend on it.

• Checking every minute gives 1,440 checks a day. At 1% you expect about 14 spurious failures a day.
• If one failure pages, that is about 14 pages a day for a service that never went down.
• If you require 3 failures in a row, and the failures are independent, the chance of a false trigger per check window is 0.01 × 0.01 × 0.01, which is one in a million. Across a day that is roughly 0.001 false pages.

Independence is the assumption doing the work. A real outage fails every check, so three in a row happens with certainty. A random blip rarely repeats three times, so it does not. Confirmation exploits that asymmetry: it costs almost nothing against real outages and removes almost all of the noise.

What it costs: detection time

The price is delay. With a check every 60 seconds and a threshold of 3, an outage that starts just after a passing check is confirmed after roughly three further checks, so expect between two and three minutes before anything fires. With a 5-minute interval the same threshold means up to 15 minutes.

That gives you a simple formula to reason with:

worst-case detection delay = interval × threshold

Decide the delay you can live with first, then split it between interval and threshold. A 3-minute budget could be a 1-minute interval with 3 failures, or a 90-second interval with 2. The first is usually better, because more frequent samples give you a finer picture of what happened.

Failures that are not independent

The arithmetic breaks when failures are correlated. Three consecutive failures caused by one cause count as one event, not three. Common cases:

• A slow dependency. If your timeout is 5 seconds and the endpoint takes 6 during a nightly job, every check in that window fails. Confirmation does not help; the timeout is simply set too tight.
• A deploy. A rolling restart can fail several checks in a row. That may be a real, short outage, or it may be planned work that should not page at all.
• A fault on the sending side. If the monitoring host loses its uplink or its resolver, all of its checks fail together and for as long as the fault lasts.

Confirmation handles random noise. It does not handle systematic causes, which need a different fix: a better timeout, a maintenance window, or a second place to check from.

Choosing the number

There is no universal threshold, but a few rules of thumb hold up.

1. Start at 3 for anything that pages a human. Two failures still leaves room for a bad moment on one path. Three is a reasonable default for a check every minute.
2. Use 1 only where a miss is more expensive than a false alert, such as a payment endpoint with a very short interval and a cheap, non-waking notification channel.
3. Scale with interval. A 10-minute interval with threshold 3 means half an hour of blindness. Lower the threshold or shorten the interval.
4. Separate the channels. A first failure can go to a log or a quiet chat channel while only the confirmed failure pages someone. The early signal is still recorded; it just does not wake anyone.

Set the timeout deliberately

Most spurious failures are timeouts, not errors. A connection refused or a 503 is a definite answer. A timeout only says that nothing arrived in time, which a slow path or a busy sender produces as readily as a dead server. Set the timeout from the endpoint's real behaviour: look at its slow tail, add margin, and let the confirmation threshold absorb the rest. A timeout copied from a default is a frequent source of noise.

Do not stop at confirmation

Confirmation decides when a monitor is down. It does not decide how often you are told afterwards. A service that flaps between up and down every few minutes will re-trigger a fresh set of failures each time, so pair the threshold with a cooldown on the notification step. Otherwise a marginal service sends a message per transition, which is the same fatigue by another route.

Also keep planned work out of the picture. Pausing a monitor for the length of a deploy is cleaner than tuning a threshold to tolerate it, and it keeps the threshold honest for the cases it is meant for.

Where Hesklo fits

In Hesklo the failure threshold is set per monitor, alongside its own timeout and interval, and notify steps carry a flap-dampening cooldown. The details are on the alert noise control page, and the check itself is described under HTTP monitoring. Pausing for planned work is covered by maintenance windows.

A checklist for one monitor

• Write down the longest detection delay you accept. Work out interval × threshold.
• Set the timeout from the endpoint's observed slow tail, not a default.
• Use a threshold of 3 for anything that wakes a person; justify any lower value.
• Add a cooldown so a flapping service does not repeat itself.
• Pause the monitor for planned work instead of absorbing it with a higher threshold.
• Route the confirmed failure into your escalation policy; send unconfirmed signals somewhere quiet.
• Review a month later: for every page, was it real? If not, find the cause rather than raising the number again.

You can try the idea without setting anything up: run a one-off request with the uptime check tool, then run it again a minute later. If the answers differ, you have just seen why one sample is not enough.