The Alert That Never Fired

Most monitoring failures are noisy. This one was the opposite: the monitoring system was working exactly as designed, reporting nothing, and nothing was precisely the problem.

What happened link

A production service degraded for 40 minutes. No alert fired. The error-rate panel on the team dashboard held a flat line the whole time. The outage was discovered when a customer reported it.

The cause was six weeks old. A migration to a shared metrics module had renamed the request counter:

  http_requests_total  →  http_server_request_duration_seconds_count
  

Roughly forty alerting rules still referenced the old name. Every one of them had been silently inert since the day the migration shipped.

Why nothing fired link

This is the part worth internalising, because it is not a bug — it is the documented semantics of the query language.

flowchart TD
    A["Metric renamed in a shared module migration"] --> B["~40 alert rules still reference the old metric name"]
    B --> C["Selector matches no series"]
    C --> D["rate() over nothing returns an empty vector"]
    D --> E["Comparison against a threshold on an empty vector is also empty"]
    E --> F["Rule evaluates cleanly and stays <b>inactive</b> — never <b>error</b>"]
    F --> G["Six weeks of silence, then a customer-reported outage"]

A typical rule from that set:

  - alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m])) > 0.05
  for: 5m
  

After the rename, http_requests_total matches no series. rate() over no series returns an empty vector, the division returns an empty vector, and comparing an empty vector to 0.05 returns an empty vector. An empty result means “no timeseries are currently breaching”, which is indistinguishable from “everything is healthy”. The rule is not broken, not erroring, not degraded — it is inactive, which is the same state it occupies on a good day.

The dashboard told the same story for the same reason. A panel querying a metric that no longer exists renders a flat line, not a gap and not an error.

The general form: in a threshold-based monitoring system, absence of data is not absence of a problem, but it looks identical. Every alert rule that depends on a metric arriving carries a second, unstated dependency — that the metric arrives at all — and nothing checks it.

Absence guards link

The runtime fix is to alert on the silence itself. Prometheus provides absent_over_time(), which returns a value only when a selector has matched nothing for the whole window:

  - alert: RequestMetricsMissing
  expr: absent_over_time(http_server_request_duration_seconds_count{job="payments-api"}[15m])
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "No request metrics from payments-api for 15m — error-rate alerting is blind"
    runbook: "https://docs.internal/runbooks/metrics-missing"
  

Two details decide whether this is useful or merely present:

• Scope the guard with an explicit selector. A bare absent_over_time(metric[15m]) tells you something stopped reporting; a guard per job tells you which service went dark. Only labels present as equality matchers in the selector survive into the alert, so the labels you want in the page have to be in the query.
• The guard is only as current as its own selector. If the next migration renames the metric again, the guard goes silent in exactly the same way the rules did. It closes the runtime gap; it does not close the change-time gap.

That second point is why the guard alone is not the fix.

Validating rules in CI link

The change-time gap closes with a check that runs when the rules or the metrics change. The critical insight is what to assert.

You cannot assert that an alert expression returns results — a healthy > 0.05 rule returns nothing, which is the entire point. What you can assert is that every metric selector in every rule matches at least one real series in a live Prometheus.

  promtool check rules alerts/**/*.yml   # syntax and structure — necessary, not sufficient
  

  #!/usr/bin/env python3
"""Fail the build if any alerting rule references a metric with no series."""
import glob, json, re, sys, urllib.parse, urllib.request, yaml

PROM = "http://prometheus.internal:9090/api/v1/query"
# ponytail: regex metric extraction, good enough for a gate. If it misfires,
# swap in a real PromQL parser rather than growing the keyword list.
IDENT = re.compile(r"\b([a-zA-Z_:][a-zA-Z0-9_:]*)\s*(?=[{\[(]|\s|$)")
NOT_METRICS = {
    "rate", "irate", "increase", "sum", "avg", "min", "max", "count", "by",
    "without", "on", "ignoring", "and", "or", "unless", "offset", "absent",
    "absent_over_time", "histogram_quantile", "clamp_max", "le", "job", "instance",
}

def has_series(metric: str) -> bool:
    q = urllib.parse.quote(f"count({metric})")
    with urllib.request.urlopen(f"{PROM}?query={q}", timeout=10) as r:
        return bool(json.load(r)["data"]["result"])

missing = []
for path in glob.glob("alerts/**/*.y*ml", recursive=True):
    for group in yaml.safe_load(open(path))["groups"]:
        for rule in group.get("rules", []):
            name = rule.get("alert") or rule.get("record")
            for metric in sorted(set(IDENT.findall(rule.get("expr", ""))) - NOT_METRICS):
                if not has_series(metric):
                    missing.append(f"{path}: {name} references {metric}")

if missing:
    print("Alert rules reference metrics with no matching series:")
    print(*missing, sep="\n")
    sys.exit(1)
print("All alert rule metrics resolve to live series.")
  

Run it in two places, because they catch different mistakes:

flowchart TD
    A[Alert rule change] --> B[CI on the pull request]
    C[Instrumentation or<br/>metric-name change] --> B
    B --> D["promtool check rules<br/>syntax and structure"]
    D --> E["Every metric selector queried<br/>against a live Prometheus"]
    E -->|no matching series| F["Build fails, naming the rule<br/>and the metric"]
    E -->|all resolve| G[Merge]
    G --> H["Scheduled re-run against production<br/>catches drift no PR touched"]
    H -->|regression| F

The scheduled run matters as much as the pull-request gate. This incident’s rename happened in a different repository from the alert rules — nothing in the metrics module’s pipeline had any reason to look at the alerting repo. A periodic check against production is what catches a break that no single change set contains.

promtool test rules is the useful complement: it feeds synthetic series into the rules and asserts which alerts fire. That verifies the logic; the query above verifies the wiring. A rule can be logically perfect and wired to nothing.

Beyond Prometheus link

The empty-vector trap is not a Prometheus quirk. Every threshold-based system has to decide what “no data” means, every one of them exposes a knob for it, and the safe setting is frequently not the default.

Check yours rather than assuming. The failure mode is identical everywhere: the system reports health because it has nothing to report, and a flat line looks like a good day.

What transfers link

Transfers to any monitoring setup, immediately:

• Absence guards on every signal a paging alert depends on. If an alert would fail to fire when its input disappears, it needs a guard. This is cheap and it is the whole mitigation.
• Treating alert rules as code with a wiring test. Syntax checks are not enough; the assertion that matters is that the metric exists.
• Running that check on a schedule, not only on pull requests. Breakage arrives from repositories that never touch the rules.
• Metric renames as breaking changes. A shared metrics module has consumers the same way an API does — alert rules, dashboards, SLO definitions, capacity reports. Rename with an overlap period: emit both names, migrate consumers, then drop the old one, exactly as in Release Orchestration and Rollback.

Worth noting about the response itself: the team’s first instinct was to fix the forty rules, which was necessary and would have left the class of failure fully intact. The guards and the CI check are what make the next rename survivable. A fix that only addresses the instance is a fix you get to repeat.

What this says about alert quality metrics: the review described in Alert Design and Noise Reduction counts pages per shift and actionable rate — both of which improved during the six weeks these rules were silent. Metrics that only measure the alerts that fire cannot see the alerts that stopped. That review now carries a coverage half asking the inverse question: for every critical user journey, which rule protects it, and when did that rule last evaluate against a non-empty result?

Adoption checklist link

• Every paging alert has an absence guard on the metrics it depends on.
• Guards carry an explicit selector so the page names the affected service.
• Alert rules live in version control and pass promtool check rules in CI.
• CI queries a live Prometheus to confirm every metric selector matches real series.
• The same check runs on a schedule, to catch breakage from other repositories.
• promtool test rules covers the firing logic of critical alerts.
• Metric renames follow an expand/migrate/contract overlap, with consumers tracked.
• The monitoring platform’s missing-data behaviour is set deliberately, not left at its default.
• Alert quality review includes coverage, not only the alerts that fired.

Case source: A 40-minute outage passed with no alert, and the error-rate panel showed a flat line.