Alert Design and Noise Reduction

Every alert is a claim that a human must stop what they are doing. Most alerting systems make that claim far too often, and the result is predictable: engineers stop reading alerts, and the one that mattered arrives in a stream of ones that did not.

The test every alert must pass link

Before an alert exists, it must answer yes to all three:

1. Is it urgent? Does it need action within minutes, rather than during the next working day? If not, it is a ticket.
2. Is it actionable? Is there something the recipient can actually do? “CPU is high” on an autoscaling service is information, not an action.
3. Is it user-visible, or reliably predictive of user impact? A single replica restarting is not worth waking anyone.

Anything failing a test becomes a dashboard entry, a ticket, or a deleted rule. Alerts that fail all three and still page are the ones that create the trust collapse described in Anti-Patterns.

Symptoms, not causes link

Cause-based alerting produces one rule per failure mode, which means the list is infinite and always missing the mode that actually occurred. Symptom-based alerting covers every cause, including the ones you never anticipated.

flowchart TD
    subgraph "Cause-based — infinite and incomplete"
    C1[Disk above 80%] --> P[Page]
    C2[Memory above 90%] --> P
    C3[Connection pool exhausted] --> P
    C4[Replica count below 3] --> P
    C5["...and the cause you<br/>did not think of"] -.->|no rule| N[No page]
    end
    subgraph "Symptom-based — complete by construction"
    S1["Checkout success rate<br/>below its SLO"] --> P2[Page]
    S2["P99 latency above<br/>its SLO"] --> P2
    P2 --> D["Diagnostics identify the cause,<br/>whatever it turns out to be"]
    end

Cause-based signals remain valuable — as dashboard panels and as diagnostic context attached to the symptom alert. They just should not page on their own.

SLO burn-rate alerting link

A threshold alert (“error rate above 1%”) fires on brief spikes that recover on their own and stays quiet during a slow degradation that consumes the entire month’s error budget. Burn-rate alerting fixes both by asking: at this rate, how fast are we consuming the error budget?

flowchart LR
    A["SLO: 99.9% success<br/>over 30 days"] --> B["Error budget:<br/>0.1% of requests"]
    B --> C{Current burn rate}
    C -->|"14.4x — budget gone in 2 days<br/>sustained 5 min + 1 h"| D["Page immediately"]
    C -->|"6x — budget gone in 5 days<br/>sustained 30 min + 6 h"| E["Page during hours"]
    C -->|"1x — budget lasts the month<br/>sustained 6 h + 3 d"| F["Ticket"]
    C -->|"below 1x"| G["No action"]

The multi-window part is what removes the noise: a fast-burn alert requires the condition to hold over both a short window (so it reacts quickly) and a longer one (so a 30-second blip does not page). The severity scales with how quickly the budget is disappearing, which is a direct proxy for how much users are suffering.

This also produces a principled answer to a question teams argue about endlessly — how bad does it have to be before we wake someone? — because the answer is derived from an availability target the business already agreed to.

Automating away the noise link

Even good alert rules produce storms when a shared dependency fails. Four mechanisms, applied in order:

Grouping. Fifty instances of the same alert become one notification listing fifty affected targets. Group by service and alert name, not by instance.

Inhibition. When a parent condition fires, suppress its children. If the database cluster is down, do not also page for every service that depends on it.

flowchart TD
    A["Database cluster unreachable"] -->|inhibits| B[payments-api error rate]
    A -->|inhibits| C[orders-api error rate]
    A -->|inhibits| D[inventory-api error rate]
    A --> E["One page: database cluster,<br/>with 3 affected services listed"]

Deduplication and throttling. Repeated notifications for an already-open alert are suppressed until it resolves or its severity changes.

Maintenance windows as data. Deployments, planned maintenance, and load tests should automatically silence the alerts they are expected to trigger — driven by the change record, not by someone remembering to click “silence”.

Enrichment: give the responder a head start link

An alert that says “payments-api error rate SLO burn 14x” starts the responder at zero. Automated enrichment can hand them the first ten minutes of investigation:

• Recent changes to this service and its dependencies, with links to the commits and deploys.
• A dependency health summary — which downstream services are also unhealthy.
• The relevant dashboard, pre-filtered to the service and time range.
• The runbook link from the service catalogue.
• Similar past incidents, matched on alert name and service.

This is one of the highest-return automations available in operations, because it applies to every single incident and requires no decision-making.

Routing link

Routing should be derived from the service catalogue, not maintained as a separate mapping that goes stale.

Escalation must be automatic: unacknowledged after N minutes goes to the secondary, then to the manager. Relying on someone noticing that nobody responded is not a plan.

Measuring alert quality link

Alerting needs its own feedback loop, or it decays.

Review these monthly, with authority to delete rules. An alert that has fired twenty times and never led to action is doing harm, and the correct response is to remove it rather than to raise its threshold slightly.

Adoption checklist link

• Every paging alert is urgent, actionable, and user-impacting.
• Alerts fire on symptoms; cause-based signals inform diagnosis only.
• SLO burn-rate alerting with multi-window conditions replaces static thresholds.
• Grouping, inhibition, deduplication, and change-driven silencing are configured.
• Alerts are automatically enriched with recent changes, dependency health, and a runbook link.
• Routing and escalation derive from the service catalogue and are automatic.
• Alert quality metrics are reviewed monthly with authority to delete rules.