Drift, State, and Immutability

Configuration drift is the gap between what your definitions say the infrastructure is and what it actually is. It accumulates silently and is usually discovered at the worst possible moment — during an incident, when someone tries to rebuild an environment and finds it does not work.

Where drift comes from link

flowchart TD
    A[Definitions in Git] -->|apply| B[Running infrastructure]
    C[Emergency console change<br/>during an incident] --> B
    D[Another tool or team<br/>managing the same resource] --> B
    E[Cloud provider changes<br/>defaults or auto-updates] --> B
    F[Manual debugging<br/>that was never reverted] --> B
    G[Autoscaling and<br/>provider-managed attributes] --> B
    B -.->|divergence grows| H["Definitions no longer<br/>describe reality"]

The emergency change is the most common and the most understandable. Someone fixes production at 3am through the console, entirely correctly, and then the change never makes it back into code. The countermeasure is not to forbid emergency changes — it is to make reconciling them a mandatory, tracked follow-up item on every incident.

Detecting drift link

Detection must be continuous and automated. A scheduled plan-only run against every environment reports the difference between definitions and reality without changing anything.

sequenceDiagram
    participant S as Scheduler (e.g. hourly)
    participant P as Plan runner (read-only role)
    participant C as Cloud
    participant A as Alerting
    S->>P: run plan for each state slice
    P->>C: read current state
    C-->>P: actual resource attributes
    P->>P: diff against definitions
    alt drift detected
        P->>A: alert with the specific diff + owning team
        Note over A: Ticket created automatically,<br/>assigned to the resource owner
    else clean
        P->>A: heartbeat "no drift"
        Note over A: Missing heartbeat is itself an alert
    end

Two design details make this work in practice:

• Use a read-only role. A drift detector with write credentials is a standing production-write path that runs unattended.
• Suppress known-dynamic attributes. Autoscaling group sizes, provider-managed tags, and last-modified timestamps produce permanent false positives. If every run reports drift, nobody reads the report — the same trust collapse described in Anti-Patterns.

Responding to drift link

Not all drift should be automatically reverted. The right response depends on the resource class:

Auto-revert is powerful and dangerous. If someone made an emergency change to keep production alive and the reconciler reverts it, you have caused an outage with automation. Restrict auto-revert to changes that are unambiguously wrong — security regressions — and alert on everything else.

Immutable infrastructure link

The structural fix for drift is to stop modifying running infrastructure at all. Instead of updating servers in place, build a new image and replace them.

flowchart LR
    subgraph "Mutable"
    M1[Server v1] -->|patch| M2[Server v1']
    M2 -->|config change| M3[Server v1'']
    M3 -->|hotfix| M4["Server v1'''<br/>state unknown, unreproducible"]
    end
    subgraph "Immutable"
    I1[Image v1] --> I2[Instances from v1]
    I3[Image v2<br/>built from source] --> I4[New instances from v2]
    I2 -.->|replaced, then terminated| I4
    I4 --> I5["Every instance identical<br/>and reproducible from the image"]
    end

What this buys:

• Drift becomes structurally impossible on the compute layer — nothing is modified, so nothing can diverge.
• Rollback is redeployment of the previous image, which is fast and reliable.
• The build is the only place changes happen, so it is the only place that needs auditing.
• Environments are genuinely identical, because they run the same image digest.

What it costs:

• Slower change cycle for small fixes: a one-line config change means a rebuild. Externalising configuration from the image mitigates this.
• Stateful workloads need explicit handling — data must live outside the replaceable unit, on network-attached storage or a managed data service.
• Longer-lived debugging sessions become harder, which is genuinely a loss; compensate with better observability rather than by re-enabling mutation.

Pets and cattle is the usual shorthand: pets are named, individually maintained, and nursed back to health; cattle are numbered, identical, and replaced when unhealthy. The practical test is whether replacing an instance is a routine operation or an event that requires planning.

State file hygiene link

For tools with explicit state, the state file is critical infrastructure and should be treated as such:

• Remote backend with locking — prevents concurrent applies from corrupting it.
• Versioning and backups — recovering from a corrupted state is otherwise a manual import exercise across hundreds of resources.
• Encryption at rest and tight access control — state files contain resource attributes including generated credentials.
• Never hand-edit. Use the tool’s own state commands (mv, rm, import), and take a backup first.
• Import rather than recreate. When a resource exists but is not in state, importing it is almost always safer than destroying and recreating it.

Reconciliation loops link

The most robust arrangement removes the need for drift detection as a separate activity: a controller continuously compares desired to actual and corrects the difference. That is the model Kubernetes uses internally and that GitOps extends to the whole delivery path — see The GitOps Operating Model.

flowchart LR
    A[Desired state<br/>in Git] --> B[Controller]
    C[Actual state<br/>in the cluster/cloud] --> B
    B --> D{Difference?}
    D -->|Yes| E[Apply correction]
    E --> C
    D -->|No| F[Wait, then re-check]
    F --> B

The loop never terminates, which is the point: drift is corrected continuously rather than discovered periodically.

Adoption checklist link

• Scheduled, read-only drift detection runs against every environment.
• Dynamic attributes are explicitly excluded so reports stay signal-rich.
• Security-relevant drift auto-reverts; other drift alerts a named owner.
• Every incident with an emergency change has a mandatory reconcile-to-code follow-up.
• Compute is immutable: instances are replaced, not patched in place.
• State is remote, locked, versioned, encrypted, and never hand-edited.
• Existing resources are imported into state rather than recreated.