Rate limiting and backpressure

• Token bucket — a bucket holds up to capacity tokens, refilled at a steady rate; each request consumes one (or more) tokens and is rejected/delayed if none are available. Naturally permits bursts up to the bucket size, then throttles to the refill rate. The common default for APIs because short bursts are usually fine.
• Leaky bucket — requests enter a fixed-size queue and are processed (leak out) at a constant rate; overflow is dropped. Smooths bursts into a perfectly steady output rate, at the cost of added latency for queued requests and dropped requests once the queue is full.
• Fixed window — count requests in discrete time windows (e.g. per calendar minute), reset the counter at each boundary. Simple, but allows 2x the intended rate at a window boundary — a burst at the end of one window plus a burst at the start of the next can both fit within the limit while occurring almost back-to-back.
• Sliding window (log or counter) — a sliding-window log tracks exact request timestamps and counts requests in the trailing window (exact but memory-heavy at scale); a sliding-window counter approximates it by weighting the previous window’s count proportionally to overlap (cheap, close approximation). Fixes the boundary-burst problem of fixed windows without the log’s memory cost.

Token bucket is the right default when bursts are acceptable; sliding window counter is the right default when precise, smooth rate enforcement matters more than burst tolerance.

• Edge / gateway / CDN — cheapest place to shed abusive or excessive load before it costs you anything downstream; typically coarse, IP- or API-key-based.
• Per-service — needed when limits are business-logic-specific (per subscription tier, per expensive operation) that the edge cannot know about, or to protect one service’s own dependency (a downstream API with its own rate limit) regardless of how traffic arrived.
• Client-side (SDK-level backoff) — advisory only, never a real control, but genuinely useful for well-behaved clients to self-throttle before even sending a request that would be rejected.

Picking only one misses real cases: edge-only limiting cannot express “this specific expensive endpoint has a tighter limit”; service-only limiting still lets abusive traffic consume network and gateway resources before being rejected. Layering is deliberate, not redundant.

A per-server in-memory counter multiplies the effective limit by the fleet size and drifts as instances scale up/down — it enforces “N requests per server,” not the intended global N. Correct approaches:

• Centralized store (Redis) with an atomic INCR + EXPIRE, or a Lua script for atomic check-and-increment to avoid a race between checking the count and incrementing it. Simple and accurate; the store becomes a shared dependency and potential hot key/bottleneck at very high throughput.
• Sharded counters — split the global limit’s counter across multiple keys/nodes and sum approximately, trading perfect accuracy for reduced contention on a single hot key.
• Local token buckets with periodic global reconciliation — each server enforces a local slice of the budget and periodically syncs with a coordinator to rebalance; resilient to a brief store outage (degrades to the last-known local budget) at the cost of being only approximately globally correct between syncs.

The right choice depends on how strictly “global” must be enforced versus how much latency and infrastructure coupling to a shared store is acceptable.

Backpressure is a signal from a slower or overloaded downstream component telling upstream callers to slow down, rather than silently queuing unboundedly or dropping work. Naive unconditional retry on failure or timeout, under load, makes things worse: every failed request gets immediately resent, multiplying load on an already-struggling system precisely when it most needs relief — a classic retry storm that can turn a transient blip into a full outage. Backpressure- aware handling instead: honor explicit signals (HTTP 429 with Retry-After, a queue reporting it is full), apply exponential backoff with jitter (randomized delay so many clients do not retry in lockstep), and — critically — have upstream callers actually reduce their own send rate or queue depth in response, not merely delay individual retries while continuing to accept new work at the same rate.

Rate limiting protects a service from being sent too much traffic; retries help an individual caller recover from a transient failure; circuit breaking protects a caller (and the struggling dependency) from continuing to hammer a dependency that is already failing. A circuit breaker tracks recent failure rate to a dependency; once it crosses a threshold, the breaker opens — subsequent calls fail fast locally without even attempting the network call — for a cooldown period, then moves to half-open (let a small trickle of requests through to test recovery) before fully closing again if they succeed. This prevents a slow or failing dependency from exhausting the caller’s own resources (threads, connections) waiting on doomed calls, and gives the struggling dependency room to recover instead of being kept at capacity by retries from every caller. The three mechanisms compose: rate limiting bounds inbound load, circuit breaking bounds outbound calls to a failing dependency, backoff-with-jitter shapes how retries are spaced.

1. Read the response — respect Retry-After if present rather than guessing a delay.
2. Back off exponentially with jitter on subsequent attempts if no explicit delay is given, to avoid synchronized retry storms across many clients.
3. Cap total retries and surface a real failure to the caller rather than retrying indefinitely — an unbounded retry loop is itself a backpressure failure waiting to happen.
4. Distinguish retryable from non-retryable failures — a 429 (rate limited) or 503 (overloaded) is retryable; a 400 (bad request) is not, and retrying it is pure wasted load.
5. Propagate the signal upward if this client is itself a service fronting other callers — reduce its own accepted concurrency or queue depth rather than absorbing the slowdown silently and letting its own upstream keep sending at full rate.
6. Add a circuit breaker if this dependency call is on a hot path, so sustained failure stops generating load entirely rather than continuing to retry-and-fail at full volume.

Plain request-count limiting is the wrong model when requests have wildly different costs (a 50-token completion vs. a 50,000-token one) — a token-bucket variant where each request consumes tokens proportional to its actual or estimated cost, rather than exactly one per request, models this correctly: the bucket holds a budget of cost-units, refilled at the allowed rate, and a request must acquire enough units (estimated up front, reconciled against actual usage afterward if the estimate was wrong) before proceeding, waiting or being rejected if the bucket cannot cover it. Practical considerations: the true cost (e.g. actual generated token count) is often only known after the request completes, so systems either reserve a conservative upper-bound estimate up front and refund the difference, or track cost in a rolling window after the fact and throttle future requests once the window’s cost budget is exhausted — and providers commonly expose the current remaining budget and reset time in response headers specifically so well-behaved clients can self-throttle without guessing.