Why Is the GPU Still a Bottleneck? 为什么 GPU 还是瓶颈？

This is the second note in my AI infrastructure learning series, following What Happens After a Prompt Enters an LLM? It is for readers who know the basics of large language models (LLMs) and want to understand the engineering limits. Numerical examples are teaching assumptions unless a source is given.

00 Picking Up the Previous Question

In Note 01, I learned why models need a key-value cache, or KV cache. It stores attention states so the model can reuse them while generating tokens. Requests with matching prefixes can also reuse compatible cached states.

But this creates a trade-off: saving computation takes memory, and GPU memory is finite.

Suppose I build an AI coding agent with an open-source 8B model. The model fits on the GPU. Why might the service still struggle when more users arrive?

Fitting the model into memory is only the first part of serving users.

I need to separate four questions. How much data can the GPU store? How fast can it move data? How much computation can it do? How does the service schedule competing requests?

01 What Uses GPU Memory?

Start with model weights, the learned parameters. An 8B model has about eight billion parameters. BF16, a 16-bit number format, uses 2 bytes per parameter. The weights alone take about 16 billion bytes, or 14.9 GiB.

The GPU also needs KV cache for active requests. It must leave room for activations, the intermediate values computed by the model, along with temporary buffers and other runtime data.

Suppose the service has 80 GiB of usable memory. Weights take 15 GiB and runtime data takes 8 GiB. Thirty independent requests each need about 1 GiB of KV cache. Total use is 15 + 8 + 30 = 53 GiB, leaving 27 GiB. Subtracting the weights alone would suggest 65 GiB free and miss most of the request memory.

Why would one request need 1 GiB of KV cache?

That amount depends on the model. Suppose it has 32 transformer layers, 8 KV heads per layer, and 128 dimensions per head. With BF16 cache values, KV memory per token is approximately:

The first 2 counts the key and value states. The last 2 is the number of bytes per BF16 value. An 8,192-token context therefore needs about 1 GiB of KV cache.

These dimensions match common Llama 8B configurations. Other models, number formats, and attention designs can have very different memory needs.

The distinction I had missed was simple: requests on one model instance share weights, but independent contexts need additional KV cache. Matching prefixes may share some cached states. I cannot assume that every user sends the same context.

Could we just shorten the context?

That was my first idea. A coding agent could drop older messages or send fewer repository files. This saves memory, but it might remove information the model needs to understand dependencies between files.

Another option is to keep the context and use prefix caching. This helps when many requests share the same beginning. If users work in different repositories, they may share little project context. A single user may reuse context across turns, but the cache still has a capacity limit.

A service can combine context selection, prefix reuse, and cache eviction, which discards cached states to free space. I would judge that combination by model usefulness and user experience within the memory budget. Cache hit rate alone is too narrow a goal.

02 Capacity, Bandwidth, and Compute

When I started learning about GPUs, memory capacity, memory bandwidth, and compute throughput sounded similar. They describe different limits.

A GPU can slow down while waiting for data even if it has free memory. Adding more compute units does little for a workload that is already waiting on memory bandwidth.

Suppose one decode step with a small batch reads about 16 GB of weights from memory. At an effective bandwidth of 1 TB/s, moving those bytes alone takes at least about 16 ms under idealized conditions.

This is not a prediction of end-to-end generation time. It leaves out caching, computation, KV access, and other costs. The calculation shows how data movement can set a lower bound on execution time.

Why do prefill and decode have different bottlenecks?

Prefill processes the input tokens. Their positions are already known, so the model can compute many positions together and reuse weights across them. With a large input, compute speed often limits this stage.

Decode generates the output. A standard autoregressive model usually generates one new token per request at each step. With a small batch, that step reads a lot of model data for relatively little computation. Memory bandwidth often becomes the limit.

Arithmetic intensity measures how many mathematical operations happen per byte moved.

Higher arithmetic intensity gives compute units more work for each byte read. With low intensity, the GPU may spend more time waiting for data.

These are common patterns, not fixed rules. The model, batch size, context length, GPU computation routines called kernels, and hardware all affect the bottleneck.

03 Why can a larger batch slow down each user?

I kept returning to this question. Batching improves GPU efficiency, so I initially expected every user to get an answer faster. But total work and individual waiting time measure different things.

For a linear layer, computing one token can be written as y = xW. Stacking token representations from different requests into a matrix X gives Y = XW. The weights W now serve more computation together.

This can raise total throughput, the amount of work the service completes per second. A larger batched step still does more work than a small one, so it may take longer even while producing more tokens.

In the eight-user batch, each user receives one token in 55 ms. Each does not receive eight tokens. The service produces more tokens overall, but the gap between each user's tokens grows from 20 ms to 55 ms.

This is a latency-throughput trade-off. It does not mean every increase in batch size makes users slower. If the GPU has spare resources, a larger batch may barely increase step time. The result depends on when compute, bandwidth, memory, or scheduling becomes the limit.

Three Metrics to Keep Separate

These metrics answer different business questions. A high-volume model API may prioritize throughput and cost per token. An interactive coding assistant may prioritize TTFT and smooth streaming.

If throughput rises by 40% while user latency stays about the same, some customers benefit. Calling that result “40% faster answers” would misrepresent what changed.

04 Requests Start and Finish at Different Times

Real traffic adds another complication. User A needs 2,000 output tokens, B needs 100, and C needs 500. D arrives later.

If a fixed batch must finish together, resources may sit idle after short requests finish. D may wait for A, the longest request. Continuous batching lets the scheduler remove finished requests and admit new ones between generation steps.

The scheduler has to track more than arrival order. Every active request uses KV memory. Some requests are in prefill; others are decoding. Admitting, pausing, and removing requests changes the available resources.

Continuous batching gives the GPU more chances to do useful work. Capacity and bandwidth still limit it. When demand is high enough, the scheduler must choose which work runs first.

05 Why can a long prefill interrupt decode?

Return to the coding agent. At 9 a.m., many developers send prompts containing large repositories. Other users already have answers streaming back.

If prefill and decode share the same GPU workers, a long prefill can occupy execution resources and pause those answers. A delay that looks like a network problem may come from GPU scheduling.

Option A: Chunked Prefill

Chunked prefill splits a large prompt into smaller pieces. The scheduler can run decode between chunks. Some implementations also mix prefill and decode work in one iteration.

This can shorten pauses between output tokens. Processing a new prompt may take longer, raising TTFT. Results depend on chunk size, scheduling policy, and workload. This approach usually needs fewer architectural changes than separating GPU worker pools.

Option B: Prefill-Decode Disaggregation

Prefill-decode disaggregation runs the two stages on different GPU workers or resource pools. Prefill workers process the input context. Decode workers generate output tokens. After prefill, the service transfers the KV cache needed for generation to the decode workers.

Separating the workers can reduce direct interference between long prefills and ongoing decode. But KV transfer takes time and network bandwidth. Long prompts can create large caches. The service also needs more routing, resource allocation, and operational work.

Separate pools can also cost more. At light load, a shared worker may be simpler and avoid transfers between workers. Dedicated pools may offer more value under sustained, heavy traffic that mixes prefill and decode.

06 How would I choose as a product manager?

I first noticed two costs: chunking may delay the completion of a new prefill, while separate pools add KV transfers and deployment complexity. That is a start, but it is too little for a product decision.

I would first ask: what are users waiting for? Is the first token late? Does output pause midway? Are requests stuck in a queue? Then I would inspect traffic during the problem, rather than rely on averages for the whole day.

I would compare the current system, chunked prefill, separate pools, and a combination if needed. With one GPU pool and brief daily peaks, chunking may be the practical first test. If large prefills constantly interrupt decode, separate pools deserve testing, provided KV transfer is fast enough.

For complaints about output freezing, I would pay particular attention to p95 and p99 ITL. The p95 value covers 95% of measured intervals; p99 covers 99%. These percentiles show the slower end of the distribution. A low average can hide pauses that make the product frustrating to use.

07 From System Metrics to User Value

I now ask three questions about an optimization: which workload does it help, which bottleneck does it reduce, and what does it cost?

Suppose an engineer says, “Our change raises throughput by 40%, but latency for each user stays about the same.”

For a high-volume inference API, the same GPUs may now do more work. With enough demand, that could lower infrastructure cost per token.

For an interactive coding assistant whose users complain about pauses, the result does not show that output feels faster. It may still help the company serve more users without GPU costs growing at the same rate. That is a separate benefit to measure.

An optimization can be technically effective and still leave a particular customer's main problem unsolved.

Understand the workload, find the bottleneck, compare the costs, and measure the result users care about.

08 Connecting the Ideas

I now ask five questions about GPU performance. Does the state fit? How must data move? Which computations dominate? How do requests interfere? Which result matters to the user?

The same coding agent may need different architectures as traffic changes. I would keep measuring, comparing, and adjusting as its workload grows.

The diagrams, batch timings, and memory budgets explain mechanisms. They are not official architectures or measured benchmarks. Actual results depend on model design, numerical precision, GPU links, kernels, concurrency, and scheduling policy.

Further Reading and References

1. TNG Technology: Prefill and Decode for Concurrent Requests: The compute and bandwidth patterns of prefill and decode.
2. NVIDIA TensorRT-LLM: Disaggregated Serving: Why separate prefill and decode.
3. NVIDIA Dynamo: Disaggregated Serving: KV transfer and resource allocation trade-offs.
4. vLLM: Architecture Overview and Scheduler: An example of scheduling in an open-source inference engine.
5. Kwon et al. (2023): PagedAttention: KV cache memory management.
6. Llama 3.1 8B Configuration Example: The model dimensions used in the KV cache estimate.