Idempotency and reliability patterns

An operation is idempotent if performing it multiple times has the exact same effect as performing it once — SET x = 5 is idempotent (repeating it leaves x = 5); x += 1 is not (repeating it changes the result each time). In a single process, this is a minor stylistic concern. In a distributed system it becomes load-bearing: at-least-once delivery is the realistic default for network calls, queue messages, and retries (see the queues subtopic) — a caller genuinely cannot always tell whether a request it sent actually succeeded server-side before a timeout or connection drop, so any correct retry strategy must assume the operation might execute more than once. If the operation is not idempotent, a retry after an ambiguous failure can silently duplicate a real-world effect — charge a card twice, send a duplicate email, decrement inventory twice for one order — without ever hitting an obvious “error.” Idempotency is what makes “just retry on failure” safe rather than a landmine.

An idempotency key is a unique identifier the client generates and attaches to a request (often a UUID), representing “this specific logical operation,” which the client resends unchanged on every retry of that same logical attempt. The server, on receiving a request with an idempotency key, checks a store (a table keyed by that identifier) for whether it has already processed this exact key: if not, it performs the operation and records the key with its result; if it has, it returns the previously recorded result without re-executing the operation — the client sees a successful response either way, but the underlying charge/send/write happens at most once. This converts an inherently non-idempotent operation (like “charge $50”) into an idempotent one from the caller’s perspective, without changing what the operation fundamentally does. The store needs a reasonable retention window (long enough to cover realistic retry timeframes) and the check-and- record step must itself be atomic (a unique constraint on the key, or a transaction) or two concurrent retries can still both slip through.

A single logical operation often triggers multiple effects across different systems — charge a payment provider, write an order row, send a confirmation email, publish an event to other services — and making the database write idempotent does nothing to guarantee the email or the external API call is not duplicated on retry, since those are separate calls with their own independent failure and retry behavior. This is the general version of the idempotency problem: every side-effecting call in the chain needs its own idempotency handling (or must be made naturally idempotent, or the whole operation needs to be structured so a retry is safe end-to-end), not just the part that happens to be a database write. It is also why partial-failure handling matters as much as retry safety — if the payment succeeds but the order-write fails and the caller retries, you need the retry to not re-charge the payment while still completing the order write, which is exactly what the idempotency key pattern (scoped to the whole logical operation, checked before re-attempting the payment step specifically) is meant to prevent.

The problem: an operation needs to atomically update its own database and publish an event (to a queue, another service) about that change — but a database write and a message publish are two separate systems with no shared transaction, so a naive “write to DB, then publish” can fail between the two steps (DB write succeeds, process crashes before publishing — the event is silently lost) or publish-then-write can publish an event for a write that never actually happened. The outbox pattern solves this by writing the event into the same database, in the same transaction as the actual data change — into an “outbox” table — so the state change and the record-of-intent-to-publish are atomically consistent by construction (a standard local ACID transaction covers both). A separate, independent process then reads unpublished rows from the outbox table and actually publishes them to the message system, retrying that publish step until it succeeds (and deleting/marking the outbox row once confirmed) — since re-reading and re-publishing an outbox row is itself idempotent-safe if the downstream consumer is built to tolerate duplicate delivery, this closes the gap without needing a genuine distributed transaction across the database and the message broker.

A saga breaks a multi-step operation that spans multiple services (book a flight, reserve a hotel, charge a card — each owned by a different service/database) into a sequence of local transactions, each with a defined compensating action that can undo it if a later step in the sequence fails — e.g. if the hotel reservation succeeds but the payment charge fails, the saga runs “cancel hotel reservation” and “cancel flight booking” as compensations, rather than relying on a single all-or-nothing distributed transaction across three different databases (which, per the sharding subtopic’s cross-shard-transaction discussion, is expensive and often not even offered across genuinely independent services). This trades true atomicity (the intermediate states really did happen and were briefly visible) for operational feasibility — the composite operation eventually reaches a consistent end state (either fully complete, or fully compensated) without requiring two-phase commit across services that were never designed to coordinate that tightly. The real design work in a saga is making every compensating action itself reliable and idempotent, since the compensation steps are subject to exactly the same retry/failure concerns as the forward steps.

They solve different halves of the same problem: retry-with-backoff decides when and how many times to re-attempt a failed call so as not to make an overloaded or transient failure worse; idempotency ensures that however many times that retry logic actually ends up calling the operation, the real-world effect is as if it happened once. A well-built client combines both: attach a stable idempotency key to the logical operation once, up front (not regenerated per retry attempt — regenerating it defeats the purpose, since each retry would look like a brand-new operation to the server); retry failed/ambiguous attempts with exponential backoff and jitter; and treat a server response of “already processed, here’s the original result” (from the idempotency-key check) as an equally successful outcome to a fresh success, not as an error to handle specially. Missing either half leaves a real gap: retries without idempotency risk duplicated effects; idempotency without sensible retry behavior means the safety mechanism sits unused because the client gives up (or hammers) instead of retrying correctly.

1. Client generates one idempotency key per logical checkout attempt (e.g. derived from the cart session, or a UUID stored client-side before the first attempt) and sends it with every retry of that same attempt.
2. Server checks the idempotency store first, inside a transaction/atomic check, before doing any work — if the key is already recorded as completed, return the stored result immediately with no side effects re-run.
3. If not yet seen, record the key as “in progress” atomically (so a concurrent retry arriving during processing does not also start a fresh attempt — it should either wait or be told to retry shortly), then proceed: charge the payment (itself ideally using the same idempotency key passed through to the payment provider, which many providers support natively), write the order record, and use the outbox pattern to reliably publish a fulfillment event in the same transaction as the order write.
4. On completion, update the idempotency record to “completed” with the result, atomically with the order write, so a subsequent retry finds a clean completed record rather than an ambiguous in-progress one.
5. On genuine failure mid-way (payment succeeded, order write failed), the retry must detect that the payment already happened (via the payment provider’s own idempotency key) and not re-charge — it should resume from “complete the order write” rather than restart the whole flow from scratch, which is why tracking which step completed, not just whether the whole operation completed, matters for a multi-step flow like this one.