Error Budgets and the SRE Model

Site reliability engineering is Google’s approach to operations, documented publicly and widely adopted in modified form. Several of its ideas have become industry defaults. The one that changes organizational behaviour most is the error budget.

The problem it solves link

Development wants to ship. Operations wants stability. Traditionally this is settled by argument, seniority, or exhaustion, and the outcome varies with who is in the room.

The error budget replaces the argument with arithmetic:

flowchart TD
    A["Service level objective:<br/>99.9% of requests succeed over 30 days"] --> B["Error budget:<br/>0.1% may fail<br/>≈ 43 minutes of full downtime"]
    B --> C{Budget remaining?}
    C -->|Yes| D["Ship. Take risks.<br/>The budget exists to be spent."]
    C -->|Exhausted| E["Feature releases pause.<br/>Reliability work takes priority<br/>until the budget recovers."]
    D --> F[Consume budget through<br/>releases, experiments, incidents]
    F --> C
    E --> G[Reliability work] --> C

Two properties make this powerful. The rule is agreed in advance, so it is not renegotiated during the argument it was meant to settle. And the budget is meant to be spent — a service consistently at 100% of its SLO is over-invested in reliability and is shipping too slowly.

Choosing the objective link

The most common mistake is setting the target too high. Each additional nine costs disproportionately more, and beyond a certain point users cannot perceive the difference because their own network and devices are less reliable than your service.

The right target is derived from what users actually need and what the business will pay for, not from what sounds impressive. Setting 99.99% on a service that cannot meet it produces a permanently exhausted budget, at which point the mechanism stops functioning and everyone ignores it.

Two related practices matter as much as the number:

• Measure from the user’s perspective. Server-side success rates miss the failures users experience at the edge, and a service can be “up” while being useless.
• Different objectives for different journeys. Checkout and the account settings page do not need the same target, and treating them identically over-invests in one and under-invests in the other.

What this automates link

The error budget is a governance mechanism that becomes automation once instrumented:

flowchart LR
    A[SLI measured continuously<br/>from real user traffic] --> B[Budget consumption calculated]
    B --> C[Burn-rate alerting<br/>pages by severity]
    B --> D[Dashboard: remaining budget<br/>visible to everyone]
    B --> E{Budget exhausted?}
    E -->|Yes| F["Release policy changes automatically:<br/>feature deploys require explicit<br/>override with justification"]
    B --> G[Input to the change<br/>risk model]

The burn-rate alerting that falls out of this is described in Alert Design and Noise Reduction. It is worth noting that this gives you a principled answer to the question every on-call rotation argues about — how bad does it have to be before we wake someone — derived from a target the business already agreed to.

Toil budgets link

The second widely-transferable idea. SRE practice caps the proportion of an SRE’s time spent on toil — commonly at 50% — with the remainder reserved for engineering work that reduces future toil.

flowchart TD
    A[Measure toil as a share<br/>of team capacity] --> B{Above the cap?}
    B -->|Yes| C[Automation work takes priority<br/>until it comes back under]
    B -->|No| D[Continue planned engineering]
    C --> E[Toil reduced] --> A
    D --> A

Without a cap, operational load expands to consume all available capacity — and a team fully consumed by operations cannot build the automation that would reduce it. The cap is what interrupts that loop. It is also the mechanism behind the practice of an SRE team being able to hand a service back to its development team when its operational burden exceeds the agreed level: a consequence that makes reliability a shared concern rather than someone else’s job.

Blameless postmortems link

Every significant incident produces a written analysis focused on systemic causes rather than individual error. This is a precondition for everything else in the model, and it is cultural rather than technical.

The mechanism: if people are punished for incidents, they conceal information, and an organization that cannot see its failures cannot improve. Blamelessness is not leniency — it is the recognition that a system which allows one person’s mistake to cause an outage has a design problem, and the design problem is the actionable finding.

What makes postmortems useful rather than ritual:

• Action items with owners and deadlines, tracked to completion. Postmortems whose actions are never done are a documentation exercise.
• A searchable archive. The value compounds only if the next person can find the previous analysis.
• A trigger threshold, so they happen consistently rather than when someone feels it was bad enough.
• Analysis of near-misses too, which are free lessons.

Preconditions link

SRE is frequently adopted as a job title without the supporting structure, which produces an operations team with a new name.

What the model actually requires:

• Measurement. SLIs from real user traffic. Without them, error budgets are fiction.
• Authority. The error budget policy must have real consequences. If feature work continues regardless, the budget is a dashboard, not a control.
• Agreement in advance, at a level senior enough to hold when it is inconvenient.
• A blameless culture, or incident data becomes unreliable.
• Enough scale to justify it. A five-person team does not need a separate reliability function; it needs the practices — SLOs, postmortems, toil awareness — without the organizational structure.

What transfers to a smaller organization link

Transfers well, at any size:

• SLOs measured from the user’s perspective.
• Error budgets as an explicit, pre-agreed trade-off.
• Burn-rate alerting instead of static thresholds.
• Blameless postmortems with tracked action items.
• Treating toil as a measured, capped quantity.

Needs adaptation:

• The separate SRE team structure. Below a certain size, embedding the practices in the product team works better than creating a function.
• The handback mechanism, which requires two teams to exist.
• Google’s specific tooling, which was built for Google’s specific scale.

The practices are separable from the org chart, and the practices are the part that produces the benefit.

Adoption checklist link

• SLIs are measured from real user traffic, per critical journey.
• SLO targets are derived from user need, not from aspiration.
• Error budget policy is agreed in advance, with real consequences.
• Burn-rate alerting replaces static thresholds.
• Remaining budget is visible to both engineering and product.
• Toil is measured as a share of capacity and capped.
• Postmortems are blameless, triggered consistently, and their actions are tracked.
• The postmortem archive is searchable.