Alert Design and Noise Reduction

Every alert is a claim that a human must stop what they are doing. Most alerting systems make that claim far too often, and the result is predictable: engineers stop reading alerts, and the one that mattered arrives in a stream of ones that did not.

The test every alert must pass link

Before an alert exists, it must answer yes to all three:

1. Is it urgent? Does it need action within minutes, rather than during the next working day? If not, it is a ticket.
2. Is it actionable? Is there something the recipient can actually do? “CPU is high” on an autoscaling service is information, not an action.
3. Is it user-visible, or reliably predictive of user impact? A single replica restarting is not worth waking anyone.

Anything failing a test becomes a dashboard entry, a ticket, or a deleted rule. Alerts that fail all three and still page are the ones that create the trust collapse described in Anti-Patterns.

Symptoms, not causes link

Cause-based alerting produces one rule per failure mode, which means the list is infinite and always missing the mode that actually occurred. Symptom-based alerting covers every cause, including the ones you never anticipated.

flowchart TD
    subgraph "Cause-based — infinite and incomplete"
    C1[Disk above 80%] --> P[Page]
    C2[Memory above 90%] --> P
    C3[Connection pool exhausted] --> P
    C4[Replica count below 3] --> P
    C5["...and the cause you<br/>did not think of"] -.->|no rule| N[No page]
    end
    subgraph "Symptom-based — complete by construction"
    S1["Checkout success rate<br/>below its SLO"] --> P2[Page]
    S2["P99 latency above<br/>its SLO"] --> P2
    P2 --> D["Diagnostics identify the cause,<br/>whatever it turns out to be"]
    end

Cause-based signals remain valuable — as dashboard panels and as diagnostic context attached to the symptom alert. They just should not page on their own.

SLO burn-rate alerting link

A threshold alert (“error rate above 1%”) fires on brief spikes that recover on their own and stays quiet during a slow degradation that consumes the entire month’s error budget. Burn-rate alerting fixes both by asking: at this rate, how fast are we consuming the error budget?

flowchart LR
    A["SLO: 99.9% success<br/>over 30 days"] --> B["Error budget:<br/>0.1% of requests"]
    B --> C{Current burn rate}
    C -->|"14.4x — budget gone in 2 days<br/>sustained 5 min + 1 h"| D["Page immediately"]
    C -->|"6x — budget gone in 5 days<br/>sustained 30 min + 6 h"| E["Page during hours"]
    C -->|"1x — budget lasts the month<br/>sustained 6 h + 3 d"| F["Ticket"]
    C -->|"below 1x"| G["No action"]

The multi-window part is what removes the noise: a fast-burn alert requires the condition to hold over both a short window (so it reacts quickly) and a longer one (so a 30-second blip does not page). The severity scales with how quickly the budget is disappearing, which is a direct proxy for how much users are suffering.

This also produces a principled answer to a question teams argue about endlessly — how bad does it have to be before we wake someone? — because the answer is derived from an availability target the business already agreed to.

Automating away the noise link

Even good alert rules produce storms when a shared dependency fails. Four mechanisms, applied in order:

Grouping. Fifty instances of the same alert become one notification listing fifty affected targets. Group by service and alert name, not by instance.

Inhibition. When a parent condition fires, suppress its children. If the database cluster is down, do not also page for every service that depends on it.

flowchart TD
    A["Database cluster unreachable"] -->|inhibits| B[payments-api error rate]
    A -->|inhibits| C[orders-api error rate]
    A -->|inhibits| D[inventory-api error rate]
    A --> E["One page: database cluster,<br/>with 3 affected services listed"]

Deduplication and throttling. Repeated notifications for an already-open alert are suppressed until it resolves or its severity changes.

Maintenance windows as data. Deployments, planned maintenance, and load tests should automatically silence the alerts they are expected to trigger — driven by the change record, not by someone remembering to click “silence”.

Enrichment: give the responder a head start link

An alert that says “payments-api error rate SLO burn 14x” starts the responder at zero. Automated enrichment can hand them the first ten minutes of investigation:

• Recent changes to this service and its dependencies, with links to the commits and deploys.
• A dependency health summary — which downstream services are also unhealthy.
• The relevant dashboard, pre-filtered to the service and time range.
• The runbook link from the service catalogue.
• Similar past incidents, matched on alert name and service.

This is one of the highest-return automations available in operations, because it applies to every single incident and requires no decision-making.

Routing link

Routing should be derived from the service catalogue, not maintained as a separate mapping that goes stale.

Escalation must be automatic: unacknowledged after N minutes goes to the secondary, then to the manager. Relying on someone noticing that nobody responded is not a plan.

Measuring alert quality link

Alerting needs its own feedback loop, or it decays.

Review these monthly, with authority to delete rules. An alert that has fired twenty times and never led to action is doing harm, and the correct response is to remove it rather than to raise its threshold slightly.

Coverage: the alerts that stopped link

Every metric in that table describes an alert that fired. None of them can see an alert that quietly stopped firing — and all five of them improve when a rule goes silent. Fewer pages, higher actionable rate, fewer repeats. A rotation that has gone blind looks, on this dashboard, like a rotation that has been fixed.

The failure is ordinary. A metric gets renamed, a label changes, a scrape job is retired, a recording rule is deleted. The rule referencing it does not error — in most query languages a selector matching nothing yields an empty result, and an empty result means “nothing is currently breaching”, which is exactly what a healthy system produces. See The Alert That Never Fired for a worked example that ran for six weeks before a customer reported the outage.

So alert review needs a second half that asks the inverse question — not “were these pages good?” but “is anything still watching?”

flowchart TD
    A["For each critical user journey"] --> B{"Which rule protects it?"}
    B -->|None| C["Coverage gap —<br/>the journey has no alert at all"]
    B -->|Named rule| D{"When did that rule last<br/>evaluate against a non-empty result?"}
    D -->|Never, or not since a known change| E["Silent rule —<br/>it is wired to nothing"]
    D -->|Recently| F["Covered"]
    E & C --> G["Backlog item with an owner,<br/>same as any other defect"]

Three checks, cheapest first:

The first two are automatable and belong in a pipeline. The third is a conversation, and it is the one that finds the gaps automation cannot — a checkout flow whose payment step is covered and whose confirmation email is not.

Two supporting habits make the coverage question answerable at all:

• Treat a rule’s last-fired date as data, not trivia. A paging rule that has never fired since it was written is either protecting something that never breaks or wired to nothing, and the two look identical until you check.
• Test that alerts can fire. Most alerting stacks can evaluate rules against synthetic series (promtool test rules and its equivalents). This verifies the logic; the selector check above verifies the wiring. A rule can be logically perfect and connected to a metric that no longer exists.

Add one row to the review table, and treat it with the same weight as the rest:

Adoption checklist link

• Every paging alert is urgent, actionable, and user-impacting.
• Alerts fire on symptoms; cause-based signals inform diagnosis only.
• SLO burn-rate alerting with multi-window conditions replaces static thresholds.
• Grouping, inhibition, deduplication, and change-driven silencing are configured.
• Alerts are automatically enriched with recent changes, dependency health, and a runbook link.
• Routing and escalation derive from the service catalogue and are automatic.
• Alert quality metrics are reviewed monthly with authority to delete rules.
• Coverage is reviewed alongside quality: every critical journey maps to a named rule.
• Every paging rule’s metric selectors are verified against live series in CI and on a schedule.
• Every paging alert has an absence guard on the signals it depends on.