Queues and asynchronous processing

A direct synchronous call couples the caller’s success to the callee’s availability and latency at that exact moment — if the callee is slow or down, the caller is blocked or fails immediately, and a traffic spike on the caller becomes an instantaneous traffic spike on the callee. A queue decouples them in time: the producer enqueues and moves on regardless of whether the consumer is currently up, slow, or scaled down; the consumer processes at its own sustainable rate, smoothing bursts rather than propagating them. What you give up: the producer no longer gets an immediate result (the interaction becomes async, requiring a different pattern — polling, webhooks, or a separate notification — for the caller to learn the outcome), and you take on real new failure modes: message loss if not configured durably, duplicate delivery, ordering is no longer free, and the queue itself becomes a new piece of infrastructure that can back up, fail, or become a bottleneck.

• At-most-once — a message is delivered zero or one times; if anything fails after send but before confirmed processing, it is simply lost. Simplest, rarely acceptable for anything that matters.
• At-least-once — the system retries until it gets acknowledgment, so a message may be delivered and processed more than once if the acknowledgment itself is lost or delayed (the consumer processed it but crashed before acking). This is what almost all production queue systems actually guarantee by default.
• Exactly-once — the message is processed with the effect of exactly one delivery, no duplicates, no loss. True end-to-end exactly-once delivery across independent systems is not achievable in general (it runs into the same fundamental problem as distributed consensus under failures) — what is actually achievable and commonly marketed as “exactly-once” is at-least-once delivery plus idempotent processing (see the idempotency subtopic), which produces the same observable effect as exactly-once even though the underlying delivery mechanism may retry. Some systems (Kafka transactions within its own ecosystem) offer genuine exactly-once semantics but only within that system’s boundary, not across an arbitrary external side effect like an email send or a payment call.

The practical takeaway: never design assuming a queue will hand you exactly-once for free — design the consumer to be idempotent and let the queue be at-least-once.

A dead-letter queue (DLQ) is where messages go after they have failed processing some bounded number of times, instead of being retried indefinitely. Retrying forever is the wrong default because a message that fails due to a permanent problem (malformed payload, a business-logic error that will never resolve by waiting) will never succeed no matter how many times it is retried, and letting it retry indefinitely both wastes processing capacity that legitimate messages need and can mask the fact that something is broken — nobody notices a silent infinite-retry loop the way they notice an alert on a growing DLQ. The DLQ turns an invisible failure into a visible, inspectable one: an operator (or an automated remediation) can look at what landed there, distinguish transient failures worth manual replay from genuinely bad messages worth discarding or fixing at the source, and the main queue keeps flowing for everything else in the meantime.

Ordering is not free and depends on the queue’s specific design and how you use it:

• Single consumer, single partition/queue — messages are typically processed in the order enqueued, since there is only one worker draining one ordered stream.
• Multiple consumers on one queue — even if the queue itself is FIFO, parallel consumers can finish processing out of enqueue order (one message takes longer than a later one that started after it), so global ordering across consumers is not preserved even though delivery order was.
• Partitioned queues (Kafka-style) — ordering is guaranteed within a partition, not across partitions; if you need ordering for a given entity (all events for one user, in order), you must route by a partition key derived from that entity so its events always land in the same partition, sacrificing cross-entity ordering (which is usually fine) for the ordering that actually matters.
• Retries break ordering trivially — a message that fails and is retried later can be processed after messages that were originally enqueued behind it.

Design for ordering deliberately (partition key choice, single-consumer-per-partition) rather than assuming a queue is inherently FIFO end-to-end.

• Pull-based (SQS-style, Kafka consumer) — the consumer requests messages when it is ready, naturally self-throttling to its own processing capacity; the broker does not need to track each consumer’s real-time readiness. Slightly higher latency (polling interval, or a long-poll wait) and puts more logic in the consumer.
• Push-based (webhooks, some pub/sub systems) — the broker delivers as soon as a message arrives, minimizing latency, but the broker must now manage backpressure itself (what happens if the consumer cannot keep up with pushes) and consumer availability tracking is more broker-side complexity.

Pull-based dominates for durable work queues specifically because consumer-controlled pacing is a simpler, more robust way to avoid overwhelming a slow or scaling-up consumer than broker-managed push backpressure.

The queue’s job here is exactly to absorb the mismatch between arrival rate (the spike) and sustainable processing rate (the downstream limit): producers enqueue all 100,000 jobs immediately (fast, decoupled from the downstream constraint entirely), and a bounded pool of consumers drains the queue at a controlled rate — enforced either by limiting consumer concurrency to match the downstream capacity, or by putting a rate limiter (token bucket, matching the downstream’s actual budget) between the consumers and the API call itself. The queue’s depth becomes the visible metric for “how far behind are we” rather than the spike causing failures — 100,000 jobs at 50/sec is about 33 minutes to drain, and that duration is a design decision to surface (an SLA, a progress indicator) rather than a problem to eliminate by brute-forcing more throughput against a hard external limit. If the downstream limit is a hard business constraint, no amount of queue or consumer scaling changes the drain time — the queue’s value is turning “this would have failed” into “this takes 33 minutes and completes reliably.”

The core risk is losing track of a message that was dequeued but never completed — if the queue marks a message “delivered” the moment a consumer picks it up and the consumer then crashes before finishing, that message is gone with no other consumer ever seeing it again unless the system accounts for this explicitly. The standard mechanism is a visibility timeout / lease: when a consumer receives a message, it becomes invisible to other consumers for a bounded window rather than being permanently removed; the consumer must explicitly acknowledge completion within that window, or the message becomes visible again for another consumer to pick up. This is also exactly why idempotent processing matters here too — a message can be picked up twice (once by the crashed consumer before it died, again after the visibility timeout expires) even though it was “processed” the first time up to the point of the crash, so the same at-least-once-plus-idempotency pattern from the delivery-guarantees question applies to crash recovery specifically, not just to network-level delivery retries.