Runbook Automation

A runbook is the answer to “what do I do about this alert?”. Most organizations have runbooks that are out of date, because prose documentation has no feedback loop — nothing fails when it drifts from reality. Executable runbooks do have one.

The progression link

flowchart LR
    A["Prose<br/><small>'restart the service'</small>"] --> B["Copy-pasteable commands<br/><small>exact commands, in order</small>"]
    B --> C["Executable script<br/><small>one command runs the sequence</small>"]
    C --> D["One-click from the alert<br/><small>parameterised, audited, safe</small>"]
    D --> E["Automatic execution<br/><small>with a human notified after</small>"]

Each step is independently valuable, and skipping to the end is a common mistake. You cannot safely automate a procedure whose failure modes you have not observed. The intermediate stages are where you find out that the procedure sometimes needs a second attempt, or that step three fails harmlessly when the cache is already empty.

Making prose runbooks less bad link

Not everything will become executable, so the prose baseline still matters:

• Start with verification, not action. “Confirm the symptom is real” prevents the responder from applying a fix to a monitoring false positive.
• Exact commands, never descriptions. kubectl -n payments rollout restart deploy/payments-api rather than “restart the payments service”.
• State the expected output. A responder who does not know what success looks like cannot tell whether the step worked.
• Include the abort criteria. “If error rate does not drop within 5 minutes, escalate to the database on-call” is the most valuable line in most runbooks.
• Say what not to do. Known-harmful actions belong in writing, because they will otherwise be attempted by someone reasoning from first principles at 3am.
• Link from the alert. A runbook that must be searched for is a runbook that will not be read.

Executable runbooks link

The step change comes from turning the procedure into code that runs in a controlled environment.

sequenceDiagram
    participant A as Alert
    participant R as Runbook runner
    participant S as Target system
    participant L as Audit log
    participant O as On-call
    A->>O: alert with "Run diagnostics" action
    O->>R: trigger (one click)
    R->>L: record who, what, when, why
    R->>S: gather state (read-only)
    S-->>R: metrics, logs, recent changes, dependency status
    R->>O: post structured summary to the incident channel
    O->>R: choose a remediation action
    R->>L: record the decision
    R->>S: execute with a scoped, short-lived credential
    S-->>R: result
    R->>O: report outcome and next step

Properties that make this safe rather than terrifying:

• Read-only diagnostics need no approval. Gathering state is harmless and should be automatic on every relevant alert — it is pure time saved.
• Every action is parameterised and validated. The runner accepts a service name from an allowlist, not an arbitrary command string.
• Everything is audited. Who ran what, when, with what arguments, and what happened.
• Credentials are short-lived and scoped to the action, not standing admin access held by the runner.
• Actions are idempotent wherever possible, so a nervous responder pressing the button twice does not compound the problem.

Diagnostics first, always link

The highest-value automation in incident response is not fixing things automatically — it is collecting the evidence automatically. Diagnostics are safe, universally applicable, and save minutes on every single incident.

A useful default diagnostic bundle, triggered on any page:

Posted into the incident channel automatically, this bundle regularly turns a twenty-minute investigation into a two-minute one.

Keeping runbooks honest link

Executable runbooks decay more slowly than prose but still decay when a procedure is used rarely.

• Test them on a schedule. Run diagnostic runbooks against production regularly — they are read-only, so this is safe, and a broken one is discovered in a test rather than an incident.
• Exercise remediation runbooks in a game day. A quarterly failure-injection exercise validates the runbook and the responder at the same time.
• Track usage. A runbook that has not been used in a year is either describing a solved problem — delete it — or an unexercised risk.
• Update from the incident. Every incident review should ask whether the runbook was right, and the fix should be part of the incident’s follow-up actions, not a separate task nobody picks up.

From runbook to permanent fix link

A runbook is a workaround with documentation. It should not be a destination.

flowchart TD
    A[Recurring incident] --> B[Write a runbook]
    B --> C[Automate the runbook]
    C --> D{Executed how often?}
    D -->|Frequently| E["Why does this keep happening?<br/>Fix the underlying cause"]
    D -->|Frequently and always the same action| F["Candidate for auto-remediation"]
    D -->|Rarely| G["Keep it; exercise it periodically"]
    E --> H[Runbook becomes unnecessary]

Frequency of runbook execution is a defect metric. A runbook run weekly is telling you about a system problem that automation is currently hiding. Automate it to stop the bleeding, then fix the cause — see Self-Healing and Auto-Remediation for how to do the first part without losing sight of the second.

Adoption checklist link

• Every paging alert links to a runbook.
• Runbooks contain exact commands, expected output, abort criteria, and known-harmful actions.
• Read-only diagnostics run automatically on every page and post to the incident channel.
• Remediation actions are parameterised, validated, audited, and idempotent.
• The runbook runner uses short-lived, scoped credentials.
• Diagnostic runbooks are tested on a schedule; remediation runbooks in game days.
• Runbook execution frequency is tracked and treated as a defect signal.