Alert fatigue is one of the most underrated reliability problems in software engineering. When every page carries the same urgency, engineers learn to tune out the noise — and that's when real incidents get missed.
Here's a practical framework for cutting through the noise.
Start with outcomes, not symptoms
The most common mistake is alerting on resource utilisation rather than user-visible outcomes. CPU at 80% might be fine or catastrophic depending on the workload. A 5% error rate is almost always catastrophic.
Always ask: "Is a user experiencing this?" If the answer is no, reconsider whether the alert should page someone at 3am.
Use the four golden signals
Google's Site Reliability Engineering book popularised four metrics that matter for almost any service: Latency, Traffic, Errors, and Saturation (LTES). Build your alerting around these before adding anything else.
Set alert windows that match your SLOs
A 1-minute spike in error rate is probably noise. A 5-minute sustained increase is worth investigating. A 15-minute window reaching your SLO error budget threshold is a crisis. Match your alert duration windows to the actual user impact timeline.
Build a tiered severity system
Not everything that fires needs to wake someone up. Consider: P1 (page immediately, customer-impacting), P2 (page during business hours), P3 (create a ticket and review weekly). Most alert fatigue comes from treating P3 issues as P1.
The goal is simple: if someone gets paged, it should require their immediate attention. If it doesn't, it's the wrong alert severity.