Chaos Engineering as a Reliability Practice

Chaos engineering is the practice of deliberately injecting failure into a system to discover weaknesses before those weaknesses discover you. It became widely known through Netflix’s Chaos Monkey — a tool that randomly terminated production instances — and the surrounding “Simian Army” of related tools, all publicly documented and largely open-sourced.

The tool is the least interesting part. What transfers is the reasoning.

The originating problem link

Netflix migrated to cloud infrastructure where individual instance failure is routine rather than exceptional. Their conclusion was that if instances will fail unpredictably, the only way to be confident the system tolerates it is to make it happen constantly, during working hours, when engineers are available to observe and fix.

The insight generalises well beyond that specific context: an untested failure path is a failure path that does not work. Retry logic, circuit breakers, failover, and graceful degradation are all code that runs only during failures. If failures are rare, that code is effectively never executed and quietly rots.

flowchart LR
    A[Failure handling code<br/>written and reviewed] --> B{Ever executed?}
    B -->|"Only during real incidents<br/>(rare, unobserved)"| C["Bugs discovered<br/>during an outage,<br/>at the worst moment"]
    B -->|"Regularly, deliberately,<br/>during working hours"| D["Bugs discovered<br/>with engineers watching<br/>and blast radius bounded"]

What the practice consists of link

Chaos engineering is a controlled experiment, not random breakage. The experimental structure is what makes it defensible:

flowchart TD
    A["1 · Define steady state<br/><small>a measurable metric of normal, e.g. stream starts per second</small>"] --> B["2 · Hypothesise<br/><small>'steady state will hold when we terminate an instance'</small>"]
    B --> C["3 · Choose the smallest<br/>meaningful blast radius"]
    C --> D["4 · Inject the failure<br/>in production or a faithful replica"]
    D --> E["5 · Measure against steady state"]
    E --> F{Hypothesis held?}
    F -->|Yes| G["Confidence increased.<br/>Widen the blast radius next time."]
    F -->|No| H["Weakness found.<br/>Stop, fix it, re-run the experiment."]

Step one carries most of the weight and is most often skipped. Without a quantitative definition of normal behaviour, an experiment produces “it seemed fine”, which is not evidence.

What makes it safe link

The reputational impression of chaos engineering — engineers breaking production for fun — is the opposite of how mature programmes run.

The rule that keeps a programme alive: stop and fix. An experiment that reveals a weakness ends the session. Continuing to inject failures into a system you know is broken is not experimentation.

Failure types worth injecting link

Latency injection is consistently the highest-yield category and the least obvious. Systems usually handle a dependency being down — the error is explicit and handled. A dependency that is slow holds connections open, exhausts thread pools, and propagates the stall upstream. Many of the largest documented outages in the industry follow that shape.

Game days link

The human counterpart to automated injection: a scheduled exercise where a team responds to an injected failure as if it were a real incident.

This tests the parts automation cannot — whether the runbook is correct, whether the on-call engineer can find it, whether escalation works, whether the diagnostic tooling is actually usable under pressure. It is also the only realistic way to exercise the rarely-run automation described in Choosing What to Automate.

A game day produces useful output when it generates a specific list: what was missing, what was slow, what was wrong in the runbook — each with an owner.

What transfers, and what does not link

Transfers to almost any organization:

• Defining steady state quantitatively before any resilience claim.
• Game days. They need no special tooling and find real problems immediately.
• Dependency failure and latency injection in staging, which is cheap and high-yield.
• The stop-and-fix discipline.

Requires more maturity:

• Production experimentation, which needs good observability, fast rollback, and organizational trust before it is responsible.
• Continuous automated injection, which needs the automatic-abort machinery to exist first.

Does not transfer directly:

• Netflix’s scale meant a single terminated instance was statistically invisible to users. In a smaller estate the same action may be user-visible, so the blast radius calculus is genuinely different. Start in staging, and move to production only where redundancy is real.
• The organizational context — high autonomy, deep observability investment, a strong blameless culture — was a precondition, not a result. Injecting failure into an organization that punishes incidents produces concealment, not learning.

Where to start link

1. Run a tabletop exercise. No injection at all: walk through “the primary database is unavailable” and see how far the team gets. This costs an hour and usually finds several gaps.
2. Inject a dependency failure in staging. Make a downstream service return errors, then make it slow. The slow case will find more.
3. Run a game day on a real service in a lower environment, with the on-call engineer responding using the actual runbook.
4. Move to production with the smallest possible blast radius, announced, in business hours, with automatic abort.
5. Automate what has passed repeatedly, and expand scope gradually.

Adoption checklist link

• Steady state is defined quantitatively before any experiment.
• Every experiment has a hypothesis and the smallest meaningful blast radius.
• Automatic abort on steady-state deviation is implemented before production experiments.
• A kill switch exists and every engineer knows it.
• Discovered weaknesses stop the session and get fixed before the next run.
• Latency and slow-dependency injection are included, not just termination.
• Game days exercise runbooks, escalation, and rarely-used automation.
• The organization’s incident culture is blameless before production experimentation begins.