Fatigue is a property of the alerting design, so it can be fixed by changing the design. This post is a method for doing that with data you already have: last month's pages.

Start with a rule for what may page a person

Before touching any monitor, write down the test an alert must pass to interrupt someone. A workable one has three parts:

1. Something is wrong now, or will be soon enough that waiting costs something. Not "something was odd a minute ago".
2. A human can do something about it. If the right response is "wait and see", a person is the wrong recipient.
3. It cannot wait until morning, or the person receiving it is already working.

Anything that fails the test is not necessarily useless. It is just not a page. It may belong in a chat channel, a daily summary, or a log. The audit below is mostly the work of sorting alerts into those buckets honestly.

The audit: label every page from the last month

Export or list every alert that reached a person in the last 30 days. For each one, record four things: which monitor, what time, what the responder did, and how long it took. Then give each a label.

• Actionable. A real problem, and someone did something that changed the outcome.
• Real but unactionable. The problem was real, but nobody could or needed to act: a provider outage you could only watch, or a blip that fixed itself before anyone opened a laptop.
• Planned. A deploy, reboot or migration that you caused on purpose.
• False. The service was fine. The check, the network on the sending side, or a threshold was wrong.
• Duplicate. The same incident delivered again, or to several people, or through several channels.

Be strict. "It was real, but it recovered" goes under unactionable, not actionable. The useful number is the share of pages that were actionable. If it is under half, the team's reaction to pages is a reasonable response to the data.

A fix for each label

False pages: raise the bar for "down"

These are the cheapest to remove, because the alert should never have existed. Require several failed checks in a row before a monitor counts as down, and set the timeout from the endpoint's real slow tail instead of a default. The arithmetic behind that is in why a check should fail three times before it pages. If failures cluster across many unrelated monitors at the same moment, suspect the sender, not the targets; a second vantage point is the cure there.

Planned pages: tell the system about the work

If a deploy pages someone, the deploy process is missing a step. Pause the affected monitors for the length of the work, with an automatic resume time so a forgotten pause does not become a blind spot for weeks. Pause only what the work touches. And do not use a pause as a cure for a monitor that is merely noisy; that hides the problem and the visibility with it. Maintenance windows cover the mechanics.

Unactionable pages: change the recipient or the channel

A real problem that nobody can fix still deserves to be recorded, but it does not deserve a 3 a.m. phone call. Move these to a quiet channel, or hold them to working hours. A branch of the policy for "important but not urgent" next to a branch for "wake someone" is usually enough. A schedule gate is one way to hold the first branch to the working day while the critical one stays open around the clock.

Duplicates: one incident, one message per destination

Count how many messages a single outage generated. If an incident that lasted ten minutes produced thirty notifications, the design is repeating itself. Make each step fire once per incident by default. Add a cooldown to services that flap, so each transition does not produce its own message. Repeats are fine when they are deliberate, for example a page that keeps going until somebody acknowledges it. That is noise you chose, and it ends when a person responds. The distinction between the two is covered on the alert noise control page.

Check the routing, not just the thresholds

Some of the worst fatigue comes from alerts that are correct but sent to everyone. A page that goes to the whole team teaches nine people to assume that someone else has it. Send the first alert to one person, and widen only if they do not respond. That is what an escalation policy that does not wake the whole team is for. Pair it with a rota so that "one person" is a named, current answer; see on-call scheduling.

Where Hesklo fits

In Hesklo the confirmation threshold and timeout are set per monitor, notify steps carry a cooldown and fire once per incident unless you ask otherwise, and pausing a monitor can resume itself at a time you set. The point of the audit is that you can apply every fix above with any tool that has these controls.

The checklist

• Write the page test down: wrong now, a human can act, cannot wait.
• Label last month's pages as actionable, unactionable, planned, false or duplicate.
• Compute the actionable share. Treat anything under half as a design defect.
• False: raise the failure threshold, fix the timeout, look for a shared cause on the sending side.
• Planned: pause the monitors for the work, with an auto-resume time.
• Unactionable: move to a quiet channel or hold to working hours.
• Duplicate: fire once per incident, add a cooldown for flapping services.
• Route the first page to one person, not the whole team.
• Repeat the audit in a month and compare the actionable share. If it did not improve, the changes were cosmetic.

Do not aim for zero alerts. Aim for a page that always means something, so the day it arrives at 3 a.m. it gets read.

The page has one job

A status page answers one question: "is it me or is it you?" It does not need to explain your architecture, apologise at length or show off your uptime. If a visitor can answer that question in five seconds, the page worked, and most of the tickets that would have followed never get written.

Two properties make that answer believable: it is current and it is specific. Most advice about wording is really advice about those two.

Let the page follow the checks, not the people

A page that someone has to remember to update is wrong exactly when it matters. During an incident the people who could update it are busy fixing the problem. The first update arrives late, the last one arrives after the fix has been quietly live for an hour, and in between the page says whatever it said before things broke.

The fix is to drive the component status from the same checks that page your team. Then "degraded" appears when your monitoring sees it, not when somebody gets a free minute. Hesklo status pages work this way: the page reads live state from the same monitors that drive your alerts. Humans still write the explanation, but they do not have to be the source of truth for the colour.

That only holds if the underlying check is trustworthy. A monitor that flaps on single timeouts will paint your public page red for a minute at a time and teach customers to ignore it. The same confirmation logic that keeps pages quiet keeps the status page honest.

What to publish

Components customers recognise

List things by what customers do, not by what you run. "Checkout", "Login", "API" and "Webhook delivery" are components. "payments-svc-eu-2" is not. Customers cannot judge whether an internal service name affects them, so they will assume the worst or ignore it.

Keep the list short. Five to ten components is plenty. Each one you add is another place the page can be wrong, and another row to read during an outage. Publishing is opt-in per monitor in Hesklo, so internal infrastructure stays private by default. That is the right default: expose a monitor only when a customer could say "that is the thing I use".

The impact, in the customer's words

Compare these two updates:

We are investigating elevated error rates on a backend dependency.

Some customers cannot log in. Already signed-in sessions are unaffected. We are investigating.

The second one lets a reader decide whether to wait or work around the problem. Say who is affected, what they see and what still works. If you do not know yet, say that too: "We do not yet know how many customers are affected" is more believable than a confident guess.

A time and a next-update promise

Put a timestamp on every update, with a timezone. End each one with when the next update will come, such as "Next update by 14:30 UTC", and then keep to it. If nothing has changed by then, post "no change, still investigating". Silence reads as abandonment, and a promised update that arrives is worth more than a vague "soon".

Planned maintenance, ahead of time

Announce planned work before it starts, with the window and what customers will notice. A monitor paused for a maintenance window should not look like an outage, and an announced window should not look like an unexplained gap.

What to leave out

• Guesses about root cause. "Looks like a database problem" becomes wrong half an hour later, and you will have to retract it in public. Describe symptoms until you are sure of the cause.
• Blame. Naming a vendor, a deploy or a person in a live update helps nobody. Customers care about their impact, not your supply chain.
• Internal jargon and hostnames. If a customer cannot act on a word, cut it.
• Security details. Do not describe the exact mechanism of an attack or a vulnerability while it is open. Say what customers should do, and publish a fuller write-up later if appropriate.
• Vanity numbers. A "99.99% uptime" badge next to a red component invites an argument you will lose.
• Optimism. Do not write "should be resolved shortly" unless you can say what "shortly" is. An ETA you miss costs more trust than no ETA.

Use the stages, and make them boring

A simple, fixed vocabulary is easier to trust than clever prose. A common sequence is:

1. Investigating. We know something is wrong. We do not know why.
2. Identified. We know the cause and are working on a fix.
3. Monitoring. A fix is in place and we are watching it.
4. Resolved. Everything has been healthy for a stated period.

Do not skip straight to "resolved" the moment a check goes green. Services often recover, relapse and recover again, especially after a restart or a cache flush. Waiting through a short "monitoring" stage costs you ten minutes of caution and saves you a second, embarrassing "we are investigating again".

Be accurate about when it started

Customers know when they first saw errors. If your page says the incident began at 14:10 and they were failing at 13:50, they conclude you either did not notice or are shaving the number. Where your detection was slower than the real start, say so: "Errors began at about 13:50 UTC; we were alerted at 14:10." It is an uncomfortable sentence and it is the one that makes the rest believable.

This is also a reason to keep monitoring honest about timing. The history of checks gives you a record of when the first failure was seen, which is better than reconstructing it from memory afterwards.

An update template

[Stage] - [Component]: [what customers experience]

Impact: [who is affected, what still works]
Started: [time, timezone]
What we are doing: [one sentence, no speculation]
Next update: [time, timezone]

A filled-in example:

Investigating - API: Requests are failing with 502 errors

Impact: Some API requests fail. The dashboard and webhooks are unaffected.
Started: 13:50 UTC
What we are doing: We are looking at recent changes and checking capacity.
Next update: 14:30 UTC

Checklist

• Component colours come from monitors, not from someone's memory.
• Components are named after what customers do, and there are fewer than ten.
• Internal-only monitors are not published.
• Every update has a time, a stage, the impact and a next-update time.
• No root-cause guesses, no blame, no jargon.
• "Resolved" waits for a short monitoring period.
• The start time reflects when customers were first affected.
• Your page lives somewhere customers will find it, ideally on your own domain such as status.your-domain.com, and is linked from your app and support footer.

None of this requires special tooling. A page that is accurate, specific and a little dull will outperform a polished one that is late. If you do want one wired to your monitors, see status pages and pricing for which plans include them.

What a single location can and cannot see

Between the monitor and your service sit the monitor's own host, its uplink, its resolver, a chain of transit networks, and finally your edge. A failure can come from any link. From one location you cannot tell which.

• The sender's trouble. A dead uplink, a broken resolver or a saturated link on the monitoring host fails every check at once, for every target.
• A bad path. One transit provider drops traffic toward your network. The service is fine for most users; the monitor happens to sit behind the broken route.
• A regional problem. A CDN node, a geo rule or a regional ISP fault breaks the service for one part of the world and not another.
• A real outage. The service is down for everyone.

Only the last one deserves a page at 03:00. The other three look identical from where the monitor stands: a timeout or a refused connection.

Retries help, but they share the vantage point

Repeating the check, as covered in our post on why an uptime check should fail three times before it pages, removes random noise. It does nothing for a systematic cause, because the retry travels the same path from the same host. If the route is broken for ten minutes, ten retries fail for ten minutes. A second location is the cheapest way to add information that a retry cannot.

Reading two opinions

Once you have a second vantage point, you need a rule for combining the answers. There are four situations when location A and location B each report:

The interesting row is the second. Disagreement is information: it says the problem is not simply "the target is dead". Whether to page on it is a policy choice, and it depends on what the monitor protects.

Choosing the policy

Require agreement to page

Page only when both locations see the failure. This removes nearly all path-related false alerts. The cost is that a genuine outage affecting only one region does not page, and you must be comfortable with that. For a service with users everywhere, a partial outage in one region is real, but it is usually a different kind of incident from "the site is down", and it often deserves a lower urgency anyway.

Page on either

Page when any location fails. You miss nothing, and you page for every bad path on the internet. Use this only where a missed regional failure is truly expensive and the notification is cheap, such as a message to a chat channel rather than a phone call.

Tier by agreement

The most useful setup splits the two cases. Agreement goes to the page-a-human path. Disagreement goes to a quiet channel with the detail attached ("failed from A, reachable from B"). The first case wakes someone; the second gets read in the morning, and if it persists it becomes a ticket.

Choose second locations carefully

• Different networks, not just different addresses. Two probes in the same datacentre share an uplink and fail together. The point is independent paths.
• Different regions. Spread across continents if your users are. A single extra probe in the same country catches little.
• Ask only when needed. You do not need continuous multi-site traffic to a healthy target. Running the second opinion only after the first location sees a failure gives you nearly the same protection at a fraction of the traffic.

Limits of the idea

A second location is not proof. Two probes can both be blocked by the same firewall rule, a rate limit or a bot filter that treats monitoring traffic as hostile. If your target blocks the probes while serving real users, every location will say "down", and you will be paged about an outage that does not exist. Allow-list your monitors, and check what your WAF does with repeated requests from the same few sources.

The reverse also holds. A second location that can reach the target does not prove users can. If the page loads for a probe but a login flow is broken, only an assertion on the response, not a second location, will show it. Treat location and content as separate questions.

Where Hesklo fits

Hesklo checks from several locations across North America, Europe and Asia, and a failure seen in one region is re-run from another before it counts as an outage. That is the "require agreement" policy, built in. The mechanics are on the features page, and the check types it applies to are described under HTTP monitoring. How to keep the remaining noise down is covered in alert noise control.

A checklist

1. For each paging monitor, write down which of the four situations above you would want to be woken for.
2. Make sure at least one other location sits on a different network and in a different region.
3. Decide the policy for disagreement: ignore, quiet channel or page. Do not leave it as an accident of configuration.
4. Allow-list your monitoring traffic in any firewall, WAF or rate limiter.
5. Pair location checks with a content assertion, so "reachable" does not stand in for "working".
6. Send confirmed failures into an escalation policy, not straight to the whole team.
7. After the next incident, look at what each location reported and when. That record is how you tune the policy.

To see the effect of vantage point yourself, run a URL through the uptime check tool, then compare it with a request from your own laptop. When the two disagree, you have found a path problem, which is exactly what a single monitoring location cannot show you.

Start from the failure you are protecting against

An escalation policy exists to answer one question: what happens if the person we paged does not respond? They may be asleep, in a tunnel, on a flight, or have muted the channel. The policy is the plan for that case. Everything else is routing.

Two opposite mistakes are common:

• Too wide. Everyone is paged at once. Each person assumes someone else will pick it up, and the cost of every incident is multiplied by the size of the team.
• Too narrow. One person is paged once. If that page fails, the outage runs until a customer complains.

The fix for both is the same: time. Widen the audience only as the clock runs.

Build it in tiers

A workable shape for a small team has three tiers.

1. Tier 1: the person on call. One named human, reached by the channel most likely to interrupt them. Chat is fine for a daytime warning; for a night-time outage it should be something that rings, such as a push notification, SMS or a phone call.
2. Tier 2: a backup. A second person, paged only if tier 1 has not acknowledged within the window. They are not a co-owner. They are insurance against tier 1 being unreachable.
3. Tier 3: a last resort. A team lead, a shared channel, or a ticket. This tier is rarely reached. Its job is that the alert is never only in somebody's pocket.

Three tiers is usually enough. Every extra tier adds delay and another place for the policy to be wrong.

Choose the waits deliberately

The wait between tiers is a trade between speed and noise. Too short, and you page the backup while tier 1 is still reaching for the phone. Too long, and a dead pager costs you the better part of an hour.

A reasonable starting point for a customer-facing service is 5 to 10 minutes before tier 2 and another 10 to 15 before tier 3. Work out the worst case explicitly. If detection takes 3 minutes, tier 2 waits 10 and tier 3 waits 15, then a completely unanswered page reaches the last resort after 28 minutes. Ask whether your business can live with that number. If not, shorten the waits. If it can, resist the urge to shorten them anyway.

Internal tools, batch jobs and staging environments can have far longer waits, or no escalation at all. The policy should match what an hour of downtime costs for that service.

Acknowledgement is what stops the clock

Without an acknowledgement signal, a policy cannot tell "tier 1 is handling it" from "tier 1 never saw it". It can only keep escalating on a timer, which pages the backup even when tier 1 is already working the problem. That is exactly the noise you were trying to avoid.

So the alert itself needs a way to say "I have this". The cheapest version is a link in the alert message. One tap, no login, no hunting for the incident. Once acknowledged, the escalation stops. If nobody taps it, the next tier fires. Silence should mean "nobody has seen this", never "someone probably has".

Repeat until somebody answers

A single page is easy to miss. If nothing acknowledges, repeat the page to the current tier at an interval, say every 5 minutes, before moving on. Repetition is the cheapest escalation there is: it costs no extra people.

Put a limit on it. A page repeated forever to a phone that is switched off is just noise in the log. Repeat a few times, then move to the next tier.

Keep quiet hours honest

Not every alert deserves a 3 a.m. phone call. Decide ahead of time which alerts do:

• Page at any hour: the customer-facing service is down, a payment path is failing, data is at risk.
• Hold until morning: a certificate with three weeks left, a slow-but-working endpoint, an internal dashboard.

Mixing the two on one policy is how a team ends up muting everything. Give the urgent and the non-urgent separate paths, and send the non-urgent ones to a place that is read at working hours.

Follow the rota, not a name

Hard-coding "Alex" as tier 1 works until Alex goes on holiday. A policy that points at a role, whoever is on call right now, survives people changing. That means the rota needs to be a real, current thing and not a wiki page nobody updates. Keep it small: one person on duty at a time, no overlaps, and a clear fallback when the slot is empty. Test the fallback, because it is the one that runs on the worst night.

One policy per service, not one for the company

A global policy is simple to explain and wrong for most services. The checkout API and the internal reporting job should not share a ruleset. Let each monitor carry its own policy, and start new monitors from a copy of a sensible default rather than from nothing.

How this looks in Hesklo

In Hesklo each monitor has its own canvas, and the escalation policy is what you draw on it: a down port connected to notify steps, with waits, branches and repeats between them. Email steps can follow an on-call rota, and an acknowledge link in the email stops the escalation. The escalation policies page shows the building blocks. None of the design above depends on it, though; the same three tiers work in any tool.

A checklist to review your policy

1. Can you name tier 1, tier 2 and tier 3 for every customer-facing service right now?
2. Is the worst-case time from detection to last resort written down, and acceptable?
3. Does acknowledging an alert actually stop the escalation?
4. Is repetition capped?
5. Do urgent and non-urgent alerts take different paths?
6. Does the policy follow a rota rather than a hard-coded name?
7. Have you triggered a test alert end to end in the last quarter, including the tier 2 hop?

That last question matters most. An escalation policy you have never run is a hypothesis. Run it on a quiet afternoon, with a teammate pretending to be asleep, and fix whatever turns out to be wrong. Related reading: controlling alert noise keeps tier 1 from being paged for things that were never outages.

What actually breaks

A certificate carries a "not after" timestamp. Once the clock passes it, a client doing normal validation must reject the connection. There is no grace period and no partial failure.

• Browsers block the page behind an interstitial. Most visitors leave; a few click through, and you should not want that habit in your users either.
• API clients and SDKs fail the request with a verification error. Well-written ones do not retry their way around it, because that would defeat the point of TLS.
• Webhooks you send and receive stop. The receiving side drops your deliveries, and your calls to a partner with a lapsed certificate fail in your code, not theirs.
• Internal service-to-service calls break wherever mutual TLS or a private CA is in use, often far from anything you think of as "the website".
• Mobile apps that pin a certificate or its chain can fail until a new app version ships, which no server-side fix can speed up.

The common thread is that the failure is total and immediate. Nothing degrades gracefully, so there is no early symptom to notice.

Why it still happens when renewal is automated

Automatic issuance has made expiry rarer, and it has also changed who gets hurt. Certificates from ACME-based authorities are short-lived (Let's Encrypt issues 90-day certificates), and the industry is moving to shorter maximum lifetimes still. Shorter lifetimes mean more renewals, so a renewal path that works 99% of the time fails more often in absolute terms. These are the failures we see described again and again:

• The renewal job stops running. A cron entry lost in a server rebuild, a timer that was never enabled, a container that no longer ships the client. The job cannot report its own absence.
• Renewal runs but cannot validate. A DNS record was moved, a firewall rule now blocks the HTTP-01 challenge, an API token for the DNS provider expired.
• Renewal succeeds but nothing reloads. A new certificate is on disk, and the web server or load balancer is still serving the old one from memory.
• The wrong certificate is served somewhere. One node in a pool, one CDN edge, or one rarely used hostname still has the old file.
• Someone else owns it. A vendor endpoint, a partner API or an appliance with a manually installed certificate, where you cannot see the renewal process at all.

Notice that in the third and fourth cases the certificate on disk is fine. Only a check against the live endpoint, doing the same handshake a client does, sees what clients see.

Check the served certificate, not the file

The most reliable signal is the expiry date of the certificate that the endpoint presents right now. You can read it yourself:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -enddate

Two details matter. Pass -servername, or a server hosting several sites may hand you a different site's default certificate through SNI. And check every hostname that matters, not only the apex: www, api and the status or login hostnames are usually separate certificates or at least separate configuration.

A one-off command proves nothing about next month, so the real task is to run the same check on a schedule and alert on the number of days remaining.

Choosing a warning threshold

The threshold has to be longer than the longest delay between "someone is told" and "a new certificate is live and reloaded". Work it out from your own process rather than copying a number.

• Fully automated, 90-day certificates. Renewal normally happens around 30 days before expiry, so a healthy certificate should never be seen with fewer than about 30 days left. A warning at 21 days means "the renewal that should have happened did not". That leaves three weeks, plenty for a person to look.
• Manual or ticketed renewal. Add the time to get a purchase approved, validation completed and a change window. Two to four weeks is a common floor; use more if renewals need a vendor.
• Certificates you do not control. Warn early, to someone who can contact the owner, because your fix is an email, not a command.

Use two stages. The first is a low-urgency note, such as a ticket or a chat message, with weeks of slack. The second, a few days before expiry, should be a real page, because by then the automatic path has clearly failed and the cost of waiting is an outage. A single alert at seven days tends to be seen on a Friday and fixed on a Monday that never comes.

Who should receive it

Expiry warnings are the classic alert that nobody owns. They fire rarely, so people forget where they go. Send the early stage somewhere with an owner, a named person or a team queue, and make sure that destination survives staff changes. Avoid a shared mailbox that nobody reads: the warning is only useful if a human reads it within days.

The late stage should follow your normal escalation path, since it is now an incident in waiting. If your on-call process can be reached for an outage, it can be reached for "this will be an outage on Thursday".

Doing this in Hesklo

Every HTTPS monitor in Hesklo tracks its certificate expiry as part of the normal check, so there is no separate monitor to create. A branch node can test whether the certificate expires within a number of days you choose and route the flow to whichever notification you want. That makes the two-stage setup above two branches on one monitor. The details are on the SSL monitoring page, and the canvas that holds the branches is described under escalation policies.

A checklist

1. List every hostname clients connect to, including vendor and partner endpoints you depend on.
2. Check the served certificate for each one, with SNI, not the file on disk.
3. Find out how each renews and who owns it. Write the owner down.
4. Set a first warning longer than your slowest renewal path, and a second a few days before expiry.
5. Route the first to a named owner and the second to whoever is on call.
6. After any change to DNS, load balancers, CDNs or the renewal client, confirm the served expiry date moved.
7. Once a year, deliberately let a low-risk test certificate approach your warning threshold and confirm the alert arrives.

The last item is the one most teams skip. An alert that has never fired is an assumption, and an expiry warning is exactly the kind you rarely see in practice. Test it while nothing depends on the answer.

The arithmetic of a false alert

Suppose a healthy service has a 1% chance that any single check fails for reasons unrelated to the service: a dropped packet, a resolver hiccup, a slow TLS handshake. That figure is invented for illustration, but the shape of the result does not depend on it.

• Checking every minute gives 1,440 checks a day. At 1% you expect about 14 spurious failures a day.
• If one failure pages, that is about 14 pages a day for a service that never went down.
• If you require 3 failures in a row, and the failures are independent, the chance of a false trigger per check window is 0.01 × 0.01 × 0.01, which is one in a million. Across a day that is roughly 0.001 false pages.

Independence is the assumption doing the work. A real outage fails every check, so three in a row happens with certainty. A random blip rarely repeats three times, so it does not. Confirmation exploits that asymmetry: it costs almost nothing against real outages and removes almost all of the noise.

What it costs: detection time

The price is delay. With a check every 60 seconds and a threshold of 3, an outage that starts just after a passing check is confirmed after roughly three further checks, so expect between two and three minutes before anything fires. With a 5-minute interval the same threshold means up to 15 minutes.

That gives you a simple formula to reason with:

worst-case detection delay = interval × threshold

Decide the delay you can live with first, then split it between interval and threshold. A 3-minute budget could be a 1-minute interval with 3 failures, or a 90-second interval with 2. The first is usually better, because more frequent samples give you a finer picture of what happened.

Failures that are not independent

The arithmetic breaks when failures are correlated. Three consecutive failures caused by one cause count as one event, not three. Common cases:

• A slow dependency. If your timeout is 5 seconds and the endpoint takes 6 during a nightly job, every check in that window fails. Confirmation does not help; the timeout is simply set too tight.
• A deploy. A rolling restart can fail several checks in a row. That may be a real, short outage, or it may be planned work that should not page at all.
• A fault on the sending side. If the monitoring host loses its uplink or its resolver, all of its checks fail together and for as long as the fault lasts.

Confirmation handles random noise. It does not handle systematic causes, which need a different fix: a better timeout, a maintenance window, or a second place to check from.

Choosing the number

There is no universal threshold, but a few rules of thumb hold up.

1. Start at 3 for anything that pages a human. Two failures still leaves room for a bad moment on one path. Three is a reasonable default for a check every minute.
2. Use 1 only where a miss is more expensive than a false alert, such as a payment endpoint with a very short interval and a cheap, non-waking notification channel.
3. Scale with interval. A 10-minute interval with threshold 3 means half an hour of blindness. Lower the threshold or shorten the interval.
4. Separate the channels. A first failure can go to a log or a quiet chat channel while only the confirmed failure pages someone. The early signal is still recorded; it just does not wake anyone.

Set the timeout deliberately

Most spurious failures are timeouts, not errors. A connection refused or a 503 is a definite answer. A timeout only says that nothing arrived in time, which a slow path or a busy sender produces as readily as a dead server. Set the timeout from the endpoint's real behaviour: look at its slow tail, add margin, and let the confirmation threshold absorb the rest. A timeout copied from a default is a frequent source of noise.

Do not stop at confirmation

Confirmation decides when a monitor is down. It does not decide how often you are told afterwards. A service that flaps between up and down every few minutes will re-trigger a fresh set of failures each time, so pair the threshold with a cooldown on the notification step. Otherwise a marginal service sends a message per transition, which is the same fatigue by another route.

Also keep planned work out of the picture. Pausing a monitor for the length of a deploy is cleaner than tuning a threshold to tolerate it, and it keeps the threshold honest for the cases it is meant for.

Where Hesklo fits

In Hesklo the failure threshold is set per monitor, alongside its own timeout and interval, and notify steps carry a flap-dampening cooldown. The details are on the alert noise control page, and the check itself is described under HTTP monitoring. Pausing for planned work is covered by maintenance windows.

A checklist for one monitor

• Write down the longest detection delay you accept. Work out interval × threshold.
• Set the timeout from the endpoint's observed slow tail, not a default.
• Use a threshold of 3 for anything that wakes a person; justify any lower value.
• Add a cooldown so a flapping service does not repeat itself.
• Pause the monitor for planned work instead of absorbing it with a higher threshold.
• Route the confirmed failure into your escalation policy; send unconfirmed signals somewhere quiet.
• Review a month later: for every page, was it real? If not, find the cause rather than raising the number again.

You can try the idea without setting anything up: run a one-off request with the uptime check tool, then run it again a minute later. If the answers differ, you have just seen why one sample is not enough.