Design resilient pipelines — retry, circuit breaker, bulkhead

At a glance

Learning objectives

• Match each pipeline failure shape to its named pattern and its considerations
• Cap retries with backoff and know when a circuit breaker replaces them
• Use bulkheads and compensating transactions to bound blast radius

What you need

• Lesson 5's SLO (patterns exist to defend the budget)
• One component that fails messily today

Steps

1. Take the catalogue as-is — Microsoft's cloud design patterns include retry, circuit breaker, compensating transaction, CQRS, event sourcing, and bulkhead, each with problem, solution, and considerations, and the collection is vendor-neutral in practice. (from ref-cloud-design-patterns )
2. Adopt the format, not just the patterns — each pattern page states the problem/solution pair formally, the format this stage's design exercises expect, so write your pipeline's failure modes in that same pair form. (from ref-cloud-design-patterns )
3. Fit patterns to pipeline components — the catalogue is described as directly applicable to pipeline component design, so retry belongs on the extract, the breaker on the load, and the bulkhead between tenants. (from ref-cloud-design-patterns )
4. Give each pattern a budget to defend — Google's SRE vocabulary supplies the error budgets and incident management that decide when retries are safe and when they are a storm, and patterns without budgets multiply load exactly when the system is weakest. (from ref-sre-google )
5. Mind the consistency of repeated work — DDIA's consistency chapter is the reason an idempotent retry is safe and a non-idempotent one is not, so pair every retry with the transaction semantics that make replaying it harmless. (from ref-ddia )

Check yourself

• Every external call in your pipeline has a named failure policy — retry with capped attempts and backoff, breaker, or fail-fast to a DLQ.
• You can say which pattern protects which SLO, and what the pattern's own considerations page warns about.

Worked example

Failure shape             -> pattern                  -> pipeline application
transient downstream err  -> Retry                    -> re-poll API; backoff + cap
cascading overload        -> Circuit breaker          -> stop hammering a sinking DW
blast radius per tenant   -> Bulkhead                 -> separate pools per domain
partial multi-step load   -> Compensating transaction -> undo a half-applied load
read/write shape mismatch -> CQRS                     -> serving table vs ingest
rebuildable state         -> Event sourcing           -> replay log to reconstruct

Sourced from

• Azure Cloud Design Patterns
• Google SRE Books
• Designing Data-Intensive Applications (O'Reilly)

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Retries without backoff and a cap are a self-inflicted outage — the retry storm is a pattern too.
• A circuit breaker that trips silently hides a dead dependency — alert on state, not just errors.
• Bulkheads partition failure, not capacity — they bound blast radius, they do not add throughput.

Production considerations

• Write the failure policy into the component's interface — callers deserve to know what happens on failure.
• Review breaker trip counts in the weekly ops review; frequent trips mean a capacity or design problem.

Diagram

resilience patterns on a pipeline
   |
   |- extract --> flaky API    --> Retry (backoff + cap)
   |- load    --> sinking DW   --> Circuit breaker (stop hammering)
   |- tenants --> noisy neighbor--> Bulkhead (separate pools)
   |- multi-step load --> partial failure --> Compensating transaction
   |
   |- every pattern defends a budget (SLO from lesson 5)
    --> failure is an input; the pattern is the interface

Registry stats