Service architecture & LLM integration

At 10 req/s, almost any reasonable architecture works — a single process, a simple synchronous framework, minimal infrastructure ceremony; the bottleneck, if any, is development speed, not the runtime. At roughly 100–1,000 req/s, the GIL starts to matter concretely for CPU-bound work — a single process can no longer use multiple cores for computation, so you typically move to multiple worker processes (e.g., several Gunicorn/uWSGI workers, each a separate OS process with its own GIL) behind a load balancer, and I/O-bound portions benefit from asyncio or threading to avoid each worker blocking on network/DB waits. “Just add more processes” (horizontal scaling within one machine, or across machines) works remarkably far, but it stops working cleanly once you hit shared-state bottlenecks that don’t horizontally scale for free — a single relational database becoming a contention point as connection counts multiply across many workers/machines, a single Redis instance saturating, or in-memory caches that are now inconsistent/duplicated across many separate processes instead of one shared cache. At 10,000 req/s, the conversation shifts from “how many processes” to genuine distributed-systems architecture: connection pooling and read replicas for the database, a proper shared cache layer, careful backpressure and circuit-breaking so one slow dependency doesn’t cascade into a full outage, and often decomposing what used to be simple in-process function calls into asynchronous message-queue-mediated work, since synchronous request/response chains stop being tenable at that scale and latency budget.

The general microservices-vs-monolith tradeoff (independent deployability and fault isolation vs. operational complexity and network-call overhead) applies to Python as much as any language, but a few Python-specific factors weigh into the decision more heavily than in some other ecosystems: dependency conflicts — Python’s packaging story historically made it painful for a single process to use two libraries requiring incompatible versions of a shared dependency (there’s no per-import namespacing the way some ecosystems allow); splitting genuinely conflicting dependency needs into separate services/processes (each with its own isolated virtual environment) can be a legitimate, pragmatic reason to split, distinct from the usual “independent scaling/deployment” arguments. The GIL is also a real factor specifically for CPU-bound workloads: a monolith with heavy CPU-bound work embedded inside an otherwise I/O-bound service can end up needing careful process/worker-pool management just to use available cores — sometimes it’s architecturally simpler to extract that CPU-heavy piece into its own separately-scaled service (potentially even in a different runtime) than to keep fighting the GIL inside a single monolithic Python process. This isn’t a purely binary or dogmatic choice: starting monolithic with clean internal module boundaries (so a future split, if ever needed, is a refactor rather than a rewrite) is a genuinely reasonable default for many systems, splitting later only once a concrete pain point (not a fashion trend) actually demands it.

The two pieces of work have genuinely different lifecycles: the streaming response needs to yield tokens to the client incrementally as they’re generated (typically via a StreamingResponse that wraps an async generator), while the logging/evaluation work only has everything it needs once the entire response has finished generating, and critically should not hold up delivering tokens to the client while it runs. The right structure: as tokens are produced, both stream them to the client and accumulate them into a buffer; once generation completes, spawn the logging/evaluation work as a separate, independently-scheduled task (asyncio.create_task, keeping a strong reference to it, since an unreferenced fire-and-forget task can be garbage-collected mid-run) rather than awaiting it inline before the response finishes — this way, the client-facing stream completes and closes promptly, and the secondary evaluation work proceeds concurrently in the background without adding latency to what the user actually experiences.

GPU-bound model inference isn’t naturally “I/O-bound” the way a network call is — the GPU is genuinely busy computing, not idly waiting the way a socket waiting on a response is — so asyncio’s core value proposition (many things can overlap because most of them are just waiting) doesn’t directly apply to the raw computation itself. Where asyncio does genuinely help: managing the surrounding I/O — receiving many concurrent client requests, queuing them, and handling network communication — while the actual inference work is dispatched to (and awaited from) whatever serving layer manages the GPU (which itself often does batching: grouping multiple pending requests together into one larger inference call, since GPUs achieve much better throughput processing several inputs together than one at a time). asyncio mainly adds unhelpful overhead if someone naively assumes wrapping the actual GPU computation call itself in async/await makes it faster or more parallel — it doesn’t; the GPU is still one resource doing one thing at a time (or one batch at a time) regardless of how many Python coroutines are conceptually “waiting” on it. The real architecture separates concerns: async for request/response plumbing and queuing, and a purpose-built batching/scheduling layer for the actual GPU work.

A retry-on-failure loop alone doesn’t solve this. A semaphore bounds how many requests can be in flight concurrently, which helps but doesn’t directly model a token-per-minute limit (which is about total throughput over time, not concurrency). A token bucket (a classic rate-limiting algorithm: a bucket holds a capacity of “tokens,” refilled at a steady rate over time, and each request must acquire enough tokens — proportional to its expected token usage — before proceeding, waiting if the bucket is currently empty) is the more accurate model for a genuine tokens-per-minute API limit, since it naturally throttles overall throughput to match the limit rather than just capping simultaneous requests. LLM API rate limits are also often reported back in response headers (e.g., remaining quota, reset time) — a robust implementation reads and reacts to those headers dynamically rather than relying purely on a client-side estimate of the limit, which can drift out of sync with the provider’s actual enforcement.

Large model weights and embedding caches are typically held in memory for the entire lifetime of a long-running process — the concern isn’t the usual reference-leak pattern so much as making sure genuinely-intended long-lived large allocations don’t interact badly with Python’s memory management or a container’s memory limits. Practical measures: using __slots__/typed arrays (numpy) rather than plain Python object collections for large numeric data, since boxed Python objects carry far more per-element overhead than the raw data itself; being deliberate about cache eviction for embedding caches specifically (an unbounded cache growing forever is the same lru_cache-without-maxsize problem as any other unbounded cache, just applied to a much larger, model-adjacent dataset) — using a bounded LRU cache with a size limit informed by the container’s actual memory budget, not an arbitrary number; and setting the container’s memory limit with real headroom above the expected steady-state usage, accounting for the fact that Python’s own memory allocator doesn’t always return freed memory to the OS immediately, so observed process memory (as the OS/container sees it) can run higher than the sum of currently-live Python objects would suggest.