Know Before the Customer Calls

Friday, 4:40 p.m. A customer writes that your application has been "slow since this morning." You open the dashboard: everything is green. CPU peaks at 30%, memory is stable, and the uptime probe has returned 200 for three weeks without interruption. You haven't seen anything, and yet the customer is right. An hour later, you discover that 4% of requests take more than ten seconds, all on the same endpoint, all for customers whose accounts contain more than ten thousand rows. None of your probes could see it: they checked that the service responded, not that it responded correctly for everyone.

This gap is not a tooling flaw, it's a methodology flaw. A probe checks a hypothesis you formulated in advance: "is the process alive," "is the disk full." It is excellent for failures you have already encountered, and blind to all the others. Yet in production, the incidents that cost the most are almost always the ones nobody anticipated: a dependency that slows down without going down, a cache that empties, a SQL query whose execution plan changes after a version upgrade. The useful question is therefore not "do I have enough probes," but "when a novel incident occurs, does my data let me understand it without deploying code." That is exactly what the word observability means, and it is a measurable criterion, not a vendor slogan.