One monitoring location is a guess: why you need a second
A check from a single place cannot tell your outage from its own network trouble. How a second vantage point settles it, and the rules for combining two opinions.
When a check fails, it has told you exactly one thing: this request, from this machine, did not get the answer it expected. That is not the same as "the service is down", and a monitor that treats the two as equal will page you for faults that belong to the sender.
What a single location can and cannot see
Between the monitor and your service sit the monitor's own host, its uplink, its resolver, a chain of transit networks, and finally your edge. A failure can come from any link. From one location you cannot tell which.
- The sender's trouble. A dead uplink, a broken resolver or a saturated link on the monitoring host fails every check at once, for every target.
- A bad path. One transit provider drops traffic toward your network. The service is fine for most users; the monitor happens to sit behind the broken route.
- A regional problem. A CDN node, a geo rule or a regional ISP fault breaks the service for one part of the world and not another.
- A real outage. The service is down for everyone.
Only the last one deserves a page at 03:00. The other three look identical from where the monitor stands: a timeout or a refused connection.
Retries help, but they share the vantage point
Repeating the check, as covered in our post on why an uptime check should fail three times before it pages, removes random noise. It does nothing for a systematic cause, because the retry travels the same path from the same host. If the route is broken for ten minutes, ten retries fail for ten minutes. A second location is the cheapest way to add information that a retry cannot.
Reading two opinions
Once you have a second vantage point, you need a rule for combining the answers. There are four situations when location A and location B each report:
| A says | B says | Likely meaning |
|---|---|---|
| down | down | Real outage, or a fault close to the target. Page. |
| down | up | A's path or A itself. Do not page yet; keep watching. |
| up | down | B's path, or a regional problem. Worth a look, rarely a page. |
| up | up | Healthy. |
The interesting row is the second. Disagreement is information: it says the problem is not simply "the target is dead". Whether to page on it is a policy choice, and it depends on what the monitor protects.
Choosing the policy
Require agreement to page
Page only when both locations see the failure. This removes nearly all path-related false alerts. The cost is that a genuine outage affecting only one region does not page, and you must be comfortable with that. For a service with users everywhere, a partial outage in one region is real, but it is usually a different kind of incident from "the site is down", and it often deserves a lower urgency anyway.
Page on either
Page when any location fails. You miss nothing, and you page for every bad path on the internet. Use this only where a missed regional failure is truly expensive and the notification is cheap, such as a message to a chat channel rather than a phone call.
Tier by agreement
The most useful setup splits the two cases. Agreement goes to the page-a-human path. Disagreement goes to a quiet channel with the detail attached ("failed from A, reachable from B"). The first case wakes someone; the second gets read in the morning, and if it persists it becomes a ticket.
Choose second locations carefully
- Different networks, not just different addresses. Two probes in the same datacentre share an uplink and fail together. The point is independent paths.
- Different regions. Spread across continents if your users are. A single extra probe in the same country catches little.
- Ask only when needed. You do not need continuous multi-site traffic to a healthy target. Running the second opinion only after the first location sees a failure gives you nearly the same protection at a fraction of the traffic.
Limits of the idea
A second location is not proof. Two probes can both be blocked by the same firewall rule, a rate limit or a bot filter that treats monitoring traffic as hostile. If your target blocks the probes while serving real users, every location will say "down", and you will be paged about an outage that does not exist. Allow-list your monitors, and check what your WAF does with repeated requests from the same few sources.
The reverse also holds. A second location that can reach the target does not prove users can. If the page loads for a probe but a login flow is broken, only an assertion on the response, not a second location, will show it. Treat location and content as separate questions.
Where Hesklo fits
Hesklo checks from several locations across North America, Europe and Asia, and a failure seen in one region is re-run from another before it counts as an outage. That is the "require agreement" policy, built in. The mechanics are on the features page, and the check types it applies to are described under HTTP monitoring. How to keep the remaining noise down is covered in alert noise control.
A checklist
- For each paging monitor, write down which of the four situations above you would want to be woken for.
- Make sure at least one other location sits on a different network and in a different region.
- Decide the policy for disagreement: ignore, quiet channel or page. Do not leave it as an accident of configuration.
- Allow-list your monitoring traffic in any firewall, WAF or rate limiter.
- Pair location checks with a content assertion, so "reachable" does not stand in for "working".
- Send confirmed failures into an escalation policy, not straight to the whole team.
- After the next incident, look at what each location reported and when. That record is how you tune the policy.
To see the effect of vantage point yourself, run a URL through the uptime check tool, then compare it with a request from your own laptop. When the two disagree, you have found a path problem, which is exactly what a single monitoring location cannot show you.