Forty-seven alerts one morning, forty-three of them false
The morning we checked forty-seven alerts by hand and forty-three were noise, the problem became the instrument. What we built instead, and the five things a status check never sees.
By Laurent TulpanPublished 9 min read
One morning, forty-seven alerts. We went through them by hand. Forty-three were noise.
Not wrong exactly: each one described something that had genuinely happened. A momentary timeout, a restart, a slow network second. Every one defensible on its own, and together they had produced a person who no longer read them.
That was the moment the problem stopped being the monitoring and became the instrument.
The real cost of a false alert
The instinct is to treat false alerts as an annoyance and missed alerts as the serious failure. Over a few weeks it is the other way round.
A missed alert costs one incident. A stream of false alerts costs the reader. After two weeks, the person on duty dismisses the notification before reading it, because ninety percent of the time that is the correct behavior and they have other work. Then the real one arrives, into a channel nobody trusts.
The damage is not the noise. It is that the signal has stopped existing while the dashboard still shows it as green and healthy.
That is also why “add more monitoring” is usually the wrong response to an incident that was missed. More checks on the same design produce more noise, and the next incident is missed for the same reason as the last one.
The five things a status check never sees
Once we started listing what had actually got through, a pattern appeared. Every one of these is compatible with a perfectly healthy status code.
The page that renders broken. The server answers, the HTML arrives, and a stylesheet did not. The page is technically up and commercially useless.
The database that answers nothing. The application responds, the query returns an empty set, and every listing on the site is blank. Nothing in the response says so.
The certificate about to expire. Not expired, which produces an obvious failure. Expiring in four days, which produces nothing at all until the morning it produces everything.
The page that stopped being indexable. A noindex tag left in place after a staging deployment. No error, no alert, and traffic that decays over weeks with no visible cause.
The critical path that no longer completes. The site is up, the form loads, and submitting it fails at the third step. Every component is green, and no customer can reach you.
That list is not theoretical. Each item on it got past a plain uptime check on a system we were responsible for, and that is why the tool we ended up building has five layers rather than one.
What we built, and the rule we held
We built UptimeEZ, which is one of the four products we run ourselves, and the design rule was set before the code: every alert must say what to do.
Not “HTTP 500 on this URL”. Something closer to: this page returned an error, it last worked at this time, this changed since then, check this first.
That rule has an uncomfortable consequence, and it is the reason most monitoring tools do not follow it. It forces you to remove checks. If a check cannot produce an actionable message, it does not belong in the alert channel: it belongs in a report somebody reads on purpose, once a week, when they have decided to look.
The result is a tool whose dashboard is quieter than its competitors’, which is a hard thing to sell and the entire point.
Why we run it against our own clients’ systems
This matters more than any feature.
We use it on the systems we are responsible for. Which means every false alert costs us directly: our evening, our attention, our credibility with the client we then have to call about nothing.
That is the only reliable incentive to make monitoring honest. A monitoring vendor who does not carry the consequences of their own noise has no structural reason to reduce it, and noise is easy to sell because it looks like vigilance.
It is the same argument as pricing on a fixed fee rather than on time: what matters is not the promise, it is which party pays when the promise is wrong.
What this taught us that applies beyond monitoring
A signal nobody reads is not a signal. This generalizes further than alerting. A report nobody opens, a log nobody greps, a dashboard nobody visits: all of them are the same failure, and all of them look like diligence on an audit checklist.
Verification beats configuration. A backup that runs is a configuration. A backup that has been restored is a fact. The distinction is the same one, and it is the single most common gap we find on the assessments we run.
The interesting failures are the ones with no error. Anything that raises an exception will eventually be found. What survives for months is the failure that produces a perfectly valid response containing nothing useful, and finding those requires checking outcomes rather than components.
The question worth asking about your own monitoring
Not how many checks you have. This one: when was the last time somebody acted on an alert, rather than dismissing it?
If the honest answer is that most alerts get closed without action, the number of checks is not the problem and adding more will not help. The instrument needs replacing, and the first version of the replacement can be a single check that produces one message somebody reads.
Further reading
- Your backups run. Do they restore?: the same distinction, on recovery
- When one person is your only backup: who answers the alert at three in the morning
- Software we run ourselves: the four products, and what operating them teaches
- Fixed fee or time and materials: who carries the cost of being wrong
- Post-launch support: what a light watch actually covers
The questions this raises, answered straight.
Why do monitoring tools produce so many false alerts?
Because most of them check whether a server answers, and a server answering is not the same as a service working.
Anything transient, a slow network moment, a restart, a momentary timeout, produces an alert that resolves itself. Individually each is defensible. Collectively they train the person on duty to stop reading, which is the actual failure.
Is a false alert really harmful?
Yes, and more than a missed one in the medium term.
A missed alert costs you one incident. A stream of false alerts costs you the reader: after a few weeks the alerts get dismissed without being read, and the real one arrives into a channel nobody trusts. The damage is not the noise, it is the loss of the signal.
What should monitoring check beyond uptime?
Whether the thing the user came for still works:
- Does the page still render?
- Does the database behind it still return rows?
- Is the certificate still valid?
- Is the page still indexable?
- Does the critical path still complete?
A status code of 200 is compatible with all five of those being broken, which is exactly how a site can be down while every dashboard is green.
Should every alert require action?
That is the rule worth holding. An alert that does not tell you what to do is an interruption, not information.
In practice that means fewer alerts, each carrying what changed, since when, and what to check first. A tool that halves the number of alerts and doubles the number that get read has improved things, even though its dashboard looks quieter.
Recognize your situation? 20 minutes is enough to tell.
Describe what is happening on your side, and where your data and customers are. You get a plain answer on whether we can help, and a better route if there is one.

