How LLM inference works: KV cache, batching and latency explained

Why the first word takes longer than the rest, what the KV cache stores, how batching and paged attention make serving cheaper, and how speculative decoding speeds up generation.

LLM inference happens in two phases: prefill, where the model processes the whole prompt at once, and decode, where it generates one token at a time. The KV cache stores intermediate attention results so each new token does not recompute the whole prompt, and serving systems batch many users’ requests together to keep GPUs busy and costs down.

Inference is where most of the money in AI is spent, because it happens every time anyone uses a model. Understanding it explains why responses stream word by word, why long prompts are slower and costlier, and how providers serve millions of users on finite hardware.

Why does the first token take longer?

During prefill, the model reads every token of your prompt and computes attention across all of them. That work happens before the first word appears, so long prompts mean a longer wait, measured as time to first token. After that, decoding generates one token per step, which is why answers stream out steadily.

What is the KV cache?

In a transformer, each token produces keys and values used by attention. Without caching, generating each new token would recompute keys and values for every earlier token. The KV cache stores them, so each step only computes the newest token. It makes generation dramatically faster, but the cache grows with context length and the number of concurrent users, and quickly becomes the main consumer of GPU memory.

How does batching reduce cost?

A GPU generating tokens for one user is mostly waiting on memory. Serving many users together uses the same work far more efficiently. Continuous batching goes further: instead of waiting for a whole batch to finish, new requests join and finished ones leave at every step, keeping the GPU busy.

What is paged attention?

KV caches for different requests have different, unpredictable lengths, which wastes memory if space is reserved up front. PagedAttention, introduced with the vLLM serving system, manages the cache in small blocks, like pages in an operating system, reducing waste so more requests fit on the same GPU.

What is speculative decoding?

Speculative decoding uses a small, fast draft model to propose several tokens ahead, then the large model checks them in a single step and accepts the ones it agrees with. When the draft is often right, generation speeds up without changing the large model’s outputs.

Which levers reduce latency and cost?

Why does this matter if you only use APIs?

Because the same mechanics show up in your bill and your user experience. Long system prompts and huge retrieved contexts raise cost and time to first token; verbose outputs raise decode time; many sequential agent steps multiply everything. Engineers who understand inference design faster, cheaper applications. Our guide to why companies want engineers who understand models gives real examples.

How do you estimate LLM serving cost?

Estimate tokens per request, including system prompt, retrieved context and output, then multiply by requests per day. For hosted APIs, apply the provider’s per-token prices. For self-hosting, estimate tokens per second your hardware can sustain at your target latency, and how many GPUs that requires at peak load. The answer often shows that trimming prompts and outputs saves more than any infrastructure change.

Which metrics should you track for a live LLM service?

How does inference relate to model architecture?

Architecture choices affect serving: grouped-query attention shrinks the KV cache, mixture-of-experts models reduce compute per token but need memory for every expert, and long-context techniques change memory needs. See mixture of experts explained for one example.

Where can you learn LLM serving?

Program Zero covers KV-cache management, quantisation, speculative decoding, continuous batching, production serving systems and latency, throughput and cost-per-token trade-offs in its advanced LLM engineering phase, and model serving with APIs in its cloud and MLOps phase.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Chennai, Hyderabad, Bengaluru and Pune, growing cities such as Coimbatore, Kochi, Indore and Nagpur, and smaller towns such as Ranchi, Patna, Bhubaneswar and Raipur. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading