How Does an LLM Serve Thousands of Requests? LLM 怎么同时服务几千个请求？

This is the third note in my AI infrastructure learning series. The example throughout is a fictional startup building an AI coding agent with an open-source model such as Qwen. All token counts, timing examples, and dashboards are teaching assumptions, not measured benchmarks or a description of any one company's deployment. The diagrams show conceptual mechanisms rather than a complete production implementation.

00 One Model Is Not a Service

In the first two notes, I learned what happens inside one model computation and why the GPU can become a bottleneck. I knew about prefill, decode, KV cache, memory bandwidth, and batching. But then I ran into a different question.

What if our coding agent suddenly has a thousand users?

It's tempting to think the hard part is just making Qwen's forward pass faster. But a production system has to make decisions before the next forward pass begins: which request should run, where should its KV states live, what happens if memory fills up, and which model replica should receive the next user?

Making a model compute is one problem. Making thousands of users share that computation efficiently is another.

That second problem is LLM serving. My mental model shifted from a single model processing a single prompt to a system continuously allocating scarce compute and memory across many requests.

01 The Whole Request Lifecycle

Let's start with one user: “Find the bug in my Python repository and suggest a fix.” Here is the simplified path from the application to generated code.

Those two decisions are easy to mix up. A router might distribute requests across multiple independent model replicas. Once a request reaches one replica, that replica's scheduler decides when to execute it alongside other requests.

Inside an engine, the request can move between waiting and running states. It may also finish, be cancelled, or be preempted when resources are insufficient. That is why the arrows above are only the beginning of the story.

02 The Scheduler: Who Gets to Run Next?

Imagine A and B are already generating code. A new request, C, arrives with a long prompt. The engine doesn't have to wait until A and B finish their entire responses before considering C.

With continuous batching, the scheduler constructs a new execution workload across iterations. It can keep A and B decoding while admitting eligible work from C, subject to token budgets, memory limits, and the engine's scheduling policy.

The scheduler also needs a policy for choosing among waiting requests. First-Come, First-Served (FCFS) respects arrival order; a Longest Prefix Match (LPM) policy can favor requests likely to reuse more cached context. The latter may save computation, but it can also delay older requests.

I initially framed the choice as fairness versus efficiency. Then I noticed that the missing variable is actual latency: how many milliseconds do we save in prefill, and how long has the bypassed user already waited?

03 KV Cache: Not All Occupied Memory Is Equal

KV cache is both an accelerator and a constraint. It saves repeated attention-state computation, but it consumes finite GPU memory. The serving engine therefore needs to know not just how much memory is occupied, but who still needs it.

Reuse means a new request can use compatible cached prefix states instead of recomputing them. Eviction means discarding eligible inactive cached states to free space. Preemption means interrupting active execution when the engine cannot safely continue with its available resources.

The distinction matters. Reclaiming an inactive cached prefix may cause a future cache miss. Preempting a live request can delay a user who is already seeing output, and can require recomputing KV states when that request resumes.

To avoid mixing up requests, serving systems maintain logical request-to-token mappings and physical KV storage. A request's token sequence does not necessarily correspond to one contiguous region of GPU memory. Paged or block-based memory management makes allocations more flexible. PagedAttention is a well-known published example of this idea.

04 Admission, Waiting, and Preemption

There is another layer of decision-making before the GPU executes anything. If a server receives more work than it can handle, should it accept the work into a queue, let it run now, or reject it promptly?

A subtle problem appears during decode. A request that fits into memory now may need more KV capacity as it generates additional tokens. If admission is too aggressive, requests can repeatedly get retracted and rescheduled. The engine makes room, then may spend more GPU time recovering lost work.

Admitting more requests can make a system less productive if too many of them keep getting interrupted.

So admission control is partly about anticipating future resource demand, not merely checking whether memory is available at this instant.

05 GPU Workers and the Two Ways to Use Two GPUs

Once the scheduler selects work, a model worker or runner prepares the tensors, attention metadata, and KV mappings needed for GPU computation. The GPU performs the actual model operations; output tokens are sampled and returned for the next round of scheduling.

When we add a second GPU, we face an architectural choice.

Data parallelism avoids cross-GPU synchronization during ordinary independent inference, at the cost of duplicating model weights and having separate KV caches. Tensor parallelism can make an oversized model fit or speed up demanding computation, but its GPUs must coordinate, and communication can erase the benefit.

For our coding agent, I leaned toward data parallelism first: the Qwen model already fits on one GPU, and the problem is concurrent user demand. I would still measure throughput, tail latency, KV capacity, and cost before treating that choice as final.

06 Routing: The Least Busy GPU May Not Be the Fastest

Independent replicas introduce a new problem. Each GPU has its own working set of cached prefixes. If a coding agent repeatedly sends the same repository context, the replica that already processed it might save substantial prefill work.

But the cached replica might also be overloaded.

Compare three strategies:

One result surprised me: a higher cache hit rate can coexist with worse user latency.

The cache-aware router may have concentrated too much traffic on a few replicas. Less prefill computation helped, but longer queues hurt even more. It would be a mistake to celebrate the hit-rate increase without checking the full latency distribution.

This is a simplified decomposition. Real engines can overlap work, and the time to first output also involves execution scheduling, sampling, and networking. Cache-hit percentages alone cannot predict how much TTFT will improve.

07 Diagnosing a Slow Service, Not Just a Slow Model

Once I had the full system in mind, one question became much easier: what do we investigate when users say the coding agent is slow?

One of our exercises had this imaginary dashboard:

My first instinct was: “Decode is slow, so the GPU worker must be slow.” That's not precise enough.

The user sees inter-token latency (ITL), the interval between delivered output tokens. A long interval might mean the GPU forward computation itself takes longer. But it might also mean the request is waiting for its next turn, or that it was preempted and needs some KV state recomputed.

In the example above, I would first look for a relationship between preemption spikes and ITL spikes, then inspect per-iteration execution time. High KV utilization is a warning sign, not proof by itself. Likewise, high GPU utilization is not inherently bad, it matters whether the GPU is completing valuable work.

What Changes When We Limit Concurrency?

Suppose we reduce the maximum number of running requests per worker, from 32 to 24, and get these hypothetical results:

Existing users now see smoother code generation. New users wait longer. The service produces fewer output tokens each second. Is that a win?

It depends on the product. An interactive coding assistant might value smooth output enough to accept the trade-off. A background code-analysis pipeline might prefer total completed work. I would also measure goodput: how many requests meet the application's latency targets, rather than treating raw throughput as the whole answer.

08 The Mental Model I Wanted

At the beginning of this module, I pictured a request moving into a GPU, getting processed, and returning an answer. That was too linear.

Now I think of a serving engine as a set of decisions that repeat:

And every optimization has a cost somewhere else:

• Favor cached requests, and other users may wait longer.
• Admit more concurrent requests, and KV pressure may trigger repeated preemption.
• Reduce concurrent work, and streaming may become smoother while queueing gets worse.
• Split a model across GPUs, and communication may offset the benefit.
• Replicate models across GPUs, and routing determines how effectively their separate caches are used.

Workload → Bottleneck → Engineering trade-off → Customer value.

That's the chain I want to be able to explain, not just the vocabulary. If a coding-agent startup says users are waiting too long or generation keeps freezing, I now know to ask which kind of waiting, which resource is actually constrained, and what gets worse if we optimize it.

That takes me to the next question: once we understand the serving system, how does a real inference engine implement these mechanisms efficiently?

— References

1. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022): iteration-level scheduling and batching.
2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023): block-based KV memory management.
3. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024): prefill/decode interference and service objectives.
4. vLLM documentation: Automatic Prefix Caching: a public example of reusable KV prefixes and cache management.

Scope and limitations: These notes synthesize general serving concepts from public research and documentation. Implementations differ in scheduling, cache eviction, routing, batching, and preemption details. All diagrams and benchmark-like values are illustrative, not measurements or production claims. Prefix reuse requires compatible requests and cache residency; improved hit rate or GPU utilization does not guarantee improved latency.