Alerts You Don't Mute

Look at the alert channel of a team that has existed for two years. You'll find a hundred and forty messages per week, of which a hundred and thirty are acknowledged with no action taken, because everyone knows they resolve on their own. Ask an on-call engineer what they do when their phone rings at four in the morning: they look at the title, recognize a familiar name, and go back to sleep. They're statistically right — the alert is false 93% of the time. The day it's real, they'll go back to sleep too.

This is the only failure mode that really matters when it comes to alerting, and you can't fix it by adding rules. An alert is only worth something if it is believed, and credibility is a shared resource that gets used up: every notification with no action to take spends a part of it, permanently. The job, then, is to have few alerts, and to make sure each one corresponds to something a human must do immediately. Everything else — CPU thresholds, disk warnings at 70%, three-minute latency spikes — belongs on a dashboard or in a weekly report, not on a phone.