Release Orchestration and Rollback

Deployment moves code onto machines. Release orchestration is the harder problem: sequencing changes across services and data stores so the system is correct at every intermediate point, including the ones you did not plan to be in.

Rollback is a design property, not a button link

Teams often discover during an incident that rollback is not actually available: the previous artifact was garbage-collected, the database schema no longer matches it, or a message format change means in-flight messages cannot be read by the old code.

Rollback must be designed in:

The last row deserves emphasis. A rollback procedure that has not run in six months has an unknown success probability. Some teams roll back a production release deliberately once a month for exactly this reason.

Database migrations: the expand/contract pattern link

Schema changes are where rollback most often becomes impossible, because data changes are not reversible in the way code changes are. The standard solution is to never make a breaking schema change in a single step.

flowchart TD
    subgraph "Expand"
    E1[Add new column, nullable<br/>Add new table<br/>Both old and new schema valid]
    end
    subgraph "Migrate"
    M1[Deploy code that writes both<br/>old and new locations]
    M2[Backfill historical data]
    M3[Deploy code that reads new,<br/>still writes both]
    end
    subgraph "Contract"
    C1[Deploy code that uses new only]
    C2[Drop old column/table<br/>only after N-1 is retired]
    end
    E1 --> M1 --> M2 --> M3 --> C1 --> C2

The rule that makes this work: each step must be independently deployable and independently reversible. At no point is there a version of the application that cannot run against the current schema.

Practical constraints worth encoding as policy:

• Never rename a column. Add the new one, dual-write, backfill, then drop the old one in a later release.
• Never make a column NOT NULL in the same release that adds it.
• Additive-only migrations run automatically; destructive migrations require explicit approval and run separately from the deployment.
• Backfills run as throttled background jobs, not as part of a migration step that blocks a deployment and locks a table.

Coordinating multi-service releases link

The instinctive answer to “these three services must change together” is a coordinated release train. It is almost always the wrong answer: it reintroduces the batch-and-synchronise model that independent deployability was meant to eliminate, and it makes every release as risky as its riskiest component.

The alternative is to make changes independently deployable through compatibility, in a fixed order:

sequenceDiagram
    participant P as Provider service
    participant C as Consumer service
    Note over P,C: Goal: change a field's format
    P->>P: Release 1 — accept both formats,<br/>still emit the old one
    C->>C: Release 2 — send/read the new format
    Note over P,C: Both services now work<br/>regardless of deploy order
    P->>P: Release 3 — emit new format only
    P->>P: Release 4 — drop support for old format

Each release is independently revertible. The cost is more releases; the benefit is that no single release can take down the system, and no cross-team scheduling meeting is required.

Where genuine atomicity is needed, use a feature flag as the coordination point: deploy all services with the behaviour behind a flag, then flip the flag once. The flag flip is the atomic event, and it is instantly reversible.

Release trains versus continuous release link

Large batches are the risk multiplier here. If a release contains one change, a failure identifies its own cause. If it contains forty, the incident starts with a bisect. This is the mechanism behind the consistent finding that smaller, more frequent releases correlate with lower change failure rates.

Post-deployment verification link

A deployment is not finished when the rollout completes; it is finished when the system has been observed to be healthy under real traffic.

flowchart TD
    A[Rollout complete] --> B[Automated smoke tests<br/>against production]
    B --> C[Watch SLIs for a bake period]
    C --> D{Error rate, latency,<br/>saturation within bounds?}
    D -->|No| E[Automatic rollback]
    D -->|Yes| F[Mark release healthy<br/>Retire previous version]
    E --> G[Page the release owner<br/>with the failing signal]

The bake period should be long enough for slow-burning failures — memory leaks, connection pool exhaustion, cache-fill effects — to appear. For most services that is 15–60 minutes, not 60 seconds.

Emergency changes link

Every organization needs a path for urgent fixes, and every organization’s emergency path eventually becomes the normal path if it is easier than the standard one.

The way to keep that from happening:

• The emergency path uses the same pipeline with a reduced gate set, never a manual out-of-band deployment.
• It is logged and reviewed — every use generates a record explaining why the standard path was insufficient.
• Its usage rate is a metric. Frequent emergency use means the standard path is too slow, and the fix is to speed up the standard path, not to normalise the bypass.

Adoption checklist link

• Previous artifacts are retained; the rollback path is exercised on a schedule.
• All schema changes follow expand/migrate/contract; no breaking change ships in one step.
• Destructive migrations are separated from deployments and explicitly approved.
• Cross-service changes use compatibility sequencing or a flag flip, not release trains.
• Every deployment is followed by automated verification and a bake period.
• Automatic rollback triggers on SLI breach during the bake period.
• Emergency-path usage is logged, reviewed, and tracked as a metric.