SOFTWARE ARCHITECTURE / DISTRIBUTED SYSTEMS
Saga Pattern: What Actually Happens When Things Fail
A grounded look at compensation, state and failure visibility.
Key Insight: The saga pattern isn't really about the happy path — you don't need a saga to execute five steps that all succeed. It exists entirely for the moment step three fails after steps one and two already succeeded. Design it backward from that moment.
Once a business transaction spans more than one service, you lose the database transaction's ACID guarantees — there's no single COMMIT that atomically applies changes across Orders, Inventory, and Payments. The saga pattern is the standard answer: break the transaction into a sequence of local steps, each in its own service, with a defined way to undo previous steps if a later one fails.
The two flavors, and why it usually matters which you pick
A choreographed saga has each service listen for events and decide independently what to do next — Inventory reserves stock on hearing OrderPlaced, then publishes StockReserved, which Payments listens for. No single place holds the whole process. This is genuinely simple for two or three steps and genuinely hard to reason about past that — the "what's the actual flow" question requires reading every service's event handlers to answer.
An orchestrated saga has a coordinator that explicitly calls each step and explicitly handles the failure of each one. It's more code up front, but the entire business process is legible in one place, which matters enormously when someone needs to debug it at 2am or when a new step needs to be added.
My default for anything with more than two or three steps, or anything where the failure handling is non-trivial, is orchestration. The clarity is worth the extra code.
Compensation is the part teams under-design
Compensating transactions — the "undo" for each step — are not simply the reverse of the forward action, and treating them that way is where sagas go wrong. Reserving inventory and releasing inventory are not perfectly symmetric: what if the item was already partially allocated to another order in the meantime? What if the release happens twice because a retry fired? Every compensating action needs to be designed with the same rigor as the forward action, including its own idempotency guarantees.
A step that can't be meaningfully undone — money already sent to a third-party payment processor, an email already delivered — needs a different strategy: often that step gets deliberately placed last in the sequence, after every other step that *can* be compensated has already succeeded, precisely so the irreversible action is the one thing that doesn't need undoing.
State has to be visible, not just correct
The hardest part of operating a saga in production isn't the logic — it's answering "where is order #48213 right now, and why has it been stuck for ten minutes?" without that question requiring someone to read code. An orchestrated saga should persist its state explicitly: which step it's on, what succeeded, what's pending, what failed and is compensating. That state needs to be queryable by a human, ideally through a dashboard, not reconstructed from scattered log lines across five services during an incident.
[reserve stock] → [charge payment] → [confirm order] ↓ fail ↓ fail [no-op] [release stock] ← compensateTimeouts are part of the design, not an afterthought
A saga step that never responds is a stuck saga, and "never" happens more often than teams expect — a downstream service hangs, a network partition occurs, a message gets lost. Every step needs an explicit timeout and an explicit answer to "what do we do if this step hasn't responded in N seconds" — retry, compensate and abort, or escalate to a human. Leaving this undefined means the first time it happens in production, someone is improvising the answer live.
Key takeaways
Design a saga backward from its failure and compensation path, not forward from its happy path. Prefer orchestration over choreography once the process has real complexity — legibility matters more than saving a bit of code. Give compensating actions the same design rigor and idempotency guarantees as forward actions. Persist saga state so a human can see exactly where a transaction is stuck, and define explicit timeouts for every step before you need them.