Self-Healing and Auto-Remediation

Auto-remediation is the point where automation stops assisting the responder and becomes the responder. Done well it removes entire classes of page from the rotation. Done badly it turns a small problem into a large one at machine speed, while making the cause invisible.

What is safe to automate link

The dividing line is whether the correct response is known in advance and unambiguous given the detected condition.

The pattern in the left column: the action is reversible or idempotent, its blast radius is bounded, and the condition uniquely determines the response. The right column contains actions that are destructive, that depend on context the automation cannot see, or that could plausibly be the wrong call.

The control loop and its guardrails link

flowchart TD
    A[Condition detected] --> B{Confidence check:<br/>signal sustained,<br/>not a single sample?}
    B -->|No| Z[Wait and re-evaluate]
    B -->|Yes| C{Rate limit:<br/>have we already acted<br/>N times this hour?}
    C -->|Exceeded| Y[Stop. Page a human.<br/>Repeated remediation means<br/>the fix is not working.]
    C -->|Within limit| D{Blast radius:<br/>would this affect more<br/>than the allowed share?}
    D -->|Too broad| Y
    D -->|Bounded| E[Execute the action]
    E --> F[Verify the condition cleared]
    F -->|Cleared| G[Record the event.<br/>Notify, do not page.]
    F -->|Not cleared| Y
    G --> H[Increment the counter<br/>that feeds the defect review]

Four guardrails do the real work:

Rate limiting. The single most important one. If a service has been restarted five times in an hour, restarting it a sixth time is not a fix — it is a loop that hides an escalating problem. The rate limit converts an infinite loop into a page.

Blast-radius caps. A remediation may act on at most a defined share of a fleet — commonly one instance at a time, or 10% of a group. Without a cap, a bad health check can cause the automation to terminate every instance in the service, correctly and very quickly.

Confidence thresholds. Require the condition to be sustained across multiple evaluation intervals. Acting on a single sample means acting on noise.

Verification. After acting, confirm the condition actually cleared. An automation that fires and never checks is generating events, not outcomes.

Circuit breakers link

Every auto-remediation system needs a way to stop itself, and a way for a human to stop it.

• Automatic disable when the remediation’s own success rate drops below a threshold. If restarts are no longer clearing the condition, something has changed and the automation is now just churn.
• Global kill switch, reachable in seconds during an incident, that stops all automated actions. When a responder is trying to stabilise a system, automation fighting them is actively harmful.
• Incident-mode suppression. During a declared incident, automated remediation for the affected services should pause by default, so the automation does not undo a deliberate manual mitigation.

Do not let remediation hide the defect link

This is the failure mode that matters most, and it is subtle because everything looks healthy.

flowchart LR
    A["Memory leak in a service"] --> B["Pod OOMs every 6 hours"]
    B --> C["Auto-restart clears it"]
    C --> D["No page, no ticket, no signal"]
    D --> E["Leak persists for months"]
    E --> F["Traffic grows;<br/>OOM interval shortens to 20 minutes"]
    F --> G["Restart loop; now it is an outage,<br/>and nobody knew there was a problem"]

The countermeasures are straightforward but must be deliberate:

• Every remediation increments a visible counter, per service and per action type.
• Counters are reviewed on a fixed cadence — weekly or monthly — with the explicit question “what underlying defect is this hiding?”
• A threshold creates a ticket automatically. More than N remediations per week generates work for the owning team, whether or not anyone noticed.
• The remediation event is recorded on the service’s timeline, so an engineer investigating something else sees that the service has been restarting itself forty times a day.

Auto-remediation should be treated as a temporary measure with a stated lifetime. When you add one, record what the real fix would be. Some will legitimately stay forever — replacing a failed instance is not hiding anything — but the ones compensating for a defect should have an expiry review.

Building up to it link

flowchart TD
    A[Manual response, documented] --> B[Automated diagnostics<br/>read-only, no risk]
    B --> C[One-click remediation<br/>human decides, machine executes]
    C --> D{Same decision every time<br/>over many occurrences?}
    D -->|No| C
    D -->|Yes| E[Automatic with notification<br/>+ all four guardrails]
    E --> F[Monitor success rate<br/>and frequency]
    F -->|Success rate drops| C
    F -->|Frequency rises| G[Escalate to fix the cause]

The gate between “one-click” and “automatic” is evidence: you should be able to point at dozens of occurrences where a human, given the same signal, made the same call. If the human sometimes chooses differently, the condition is ambiguous and needs a better signal before it needs automation.

Adoption checklist link

• Only conditions with a known, unambiguous, bounded response are auto-remediated.
• Rate limits convert repeated remediation into a page.
• Blast-radius caps limit how much of a fleet one run can touch.
• Actions require a sustained signal, not a single sample.
• Every action verifies that the condition cleared.
• A global kill switch exists and is tested; incident mode suppresses remediation.
• Remediation counts are visible, reviewed, and generate tickets past a threshold.
• Each remediation records the underlying fix it is standing in for.