Progressive Delivery

Progressive delivery is what you get when the rollout mechanics of deployment strategies are combined with automated analysis, so that the software decides whether to continue the rollout. The human decides what “healthy” means; the system decides whether this release meets it.

The control loop link

flowchart TD
    A[New version deployed<br/>to a small traffic slice] --> B[Collect metrics<br/>from canary and baseline]
    B --> C[Statistical comparison<br/>over an observation window]
    C --> D{All metrics<br/>within tolerance?}
    D -->|Yes| E{At 100%?}
    E -->|No| F[Increase traffic share] --> B
    E -->|Yes| G[Promote: retire old version]
    D -->|No| H[Abort: shift traffic to 0%]
    H --> I[Notify owner with the<br/>specific failing metric]
    H --> J[Leave the failed version<br/>deployed but unrouted for diagnosis]

The essential property is that no human is in the loop for the common case. A rollout that requires someone to watch a dashboard for ten minutes has a throughput limit equal to that person’s attention, and it fails on nights and weekends.

Choosing analysis metrics link

Analysis quality is determined almost entirely by metric selection. A canary analysis that only checks HTTP 5xx rates will happily promote a release that returns 200 responses containing wrong data.

The business KPI is the one teams skip and the one that most often justifies the whole mechanism. A release that removes a button still returns 200s; only a funnel metric notices.

Comparison method link

Comparing the canary against a fixed threshold (“error rate must be below 1%”) produces false alarms whenever the baseline is legitimately elevated — during a traffic spike, a dependency incident, or a noisy neighbour.

Comparing the canary against a concurrently running baseline removes that whole class of noise: both versions experience the same conditions, so the only systematic difference is the code.

flowchart LR
    T[Live traffic] --> S{Split}
    S -->|"95%"| B["Baseline: v1<br/>(current production)"]
    S -->|"5%"| C["Canary: v2"]
    B --> M1[Metrics: v1]
    C --> M2[Metrics: v2]
    M1 --> A["Compare v2 against v1<br/>over the same window"]
    M2 --> A
    A --> D{"Is the difference<br/>beyond tolerance?"}

Two refinements matter in practice:

• Compare against a freshly deployed baseline, not against the long-running production pods. Long-lived instances have warm caches and JIT-optimised code, so a fresh canary looks worse purely because it is fresh.
• Require a minimum sample size before drawing a conclusion. On low-traffic services, ten requests will produce a confident and meaningless verdict; the analysis should say “insufficient data” and extend the window rather than guess.

Tuning the window link

Observation window length is a direct trade-off between rollout speed and detection reliability.

A common compromise is a short window at low traffic percentages to catch obvious breakage quickly, then longer windows at higher percentages where the sample is bigger and the exposure is greater. The final 100% step keeps the old version available for a defined period so an instant rollback remains possible.

Automatic abort must actually be automatic link

The most common way progressive delivery fails is that the abort is configured to page a human rather than to act. By the time someone acknowledges, reads the dashboard, and decides, the exposure window has been ten times longer than necessary.

Abort should:

1. Shift traffic to zero immediately — that is the mitigation.
2. Keep the failed version deployed but unrouted, so it can be inspected. Deleting it destroys the evidence.
3. Notify with the specific failing signal, not “rollout failed”. “Canary P99 latency 340ms vs baseline 95ms over 12 minutes, n=8,400” is diagnosable.
4. Record the abort as a change failure for DORA metrics — a caught failure is still a failure, and hiding it distorts the signal you rely on.

Experimentation as a delivery control link

The same traffic-splitting machinery supports A/B experiments, and it is worth being clear about the distinction:

• Canary analysis asks “is this version broken?” Short window, technical metrics, automatic decision.
• A/B experiment asks “is this version better?” Long window, business metrics, human decision, proper statistical design.

Sharing the infrastructure is efficient. Conflating the questions is not: a change can be perfectly healthy and commercially worse, and only one of those should abort a rollout.

Prerequisites link

Progressive delivery assumes things that are worth checking before investing:

• A traffic-routing layer that can split by percentage — service mesh, ingress controller, or load balancer with weighted targets.
• Metrics with low collection latency. If your monitoring aggregates on a five-minute delay, a five-minute analysis window is measuring nothing.
• Backward-compatible changes, since both versions run simultaneously against the same data store.
• Enough traffic. Below roughly a few hundred requests per minute on the path being changed, statistical comparison is not meaningful; use blue/green with a manual smoke test instead.

Adoption checklist link

• Promotion and abort decisions are automatic, not dashboard-watching.
• Analysis includes availability, latency percentiles, saturation, and at least one business KPI.
• Canary is compared to a concurrent, freshly deployed baseline.
• A minimum sample size is enforced before any verdict.
• Abort shifts traffic instantly, preserves the failed version, and reports the specific signal.
• Aborted rollouts are recorded as change failures.
• Canary analysis and A/B experimentation are kept conceptually separate.