KV Cache

What Is KV Cache?

The KV cache is the memory where a transformer language model stores the attention keys and values it has already computed for previous tokens. By reusing them, the model generates each new token without reprocessing the entire input, which is what makes token-by-token generation fast and what makes long contexts expensive in memory.

Key Takeaways

• Without a KV cache, every generated token would require recomputing attention over the whole sequence; with it, only the newest token is processed each step.
• KV cache size grows linearly with context length, and at long contexts it can rival or exceed the memory used by the model weights themselves.
• Prompt caching, the API feature that discounts repeated prompt prefixes, is KV cache reuse across requests, and it is the main cost lever for agent workloads.
• Cache-friendly prompt design, stable prefixes with variable content at the end, directly cuts both latency and spend.

How It Works

A transformer's attention mechanism lets each token look back at every earlier token. To do that, the model computes a key and a value vector for every token at every layer. During generation these do not change once computed: token 500 attends to the same keys and values for tokens 1 through 499 that were computed when those tokens were processed. Caching them is therefore pure win. The generation loop splits into two phases: prefill, where the model ingests the whole prompt in parallel and populates the cache, and decode, where it emits one token at a time, each step computing keys and values only for the newest token and reading everything else from the cache. Prefill is compute-bound and determines time to first token; decode is memory-bandwidth-bound and determines tokens per second.

The cost of this speed is memory. Cache size scales with layers, attention heads, head dimension, and above all sequence length, so a session with a very long context window can demand tens of gigabytes of cache per request. Much of modern inference engineering is KV cache management: grouped-query attention and multi-head latent attention shrink what must be stored, paged attention allocates cache in blocks so servers avoid wasting reserved memory, cache quantization stores keys and values at lower precision, and eviction schemes drop old entries when memory runs out. The waste being fought is substantial: the vLLM team measured that pre-PagedAttention serving systems lost 60-80% of KV cache memory to fragmentation and over-reservation, which its paged design cuts to near zero [1]. The PagedAttention paper from UC Berkeley reports 2-4x higher serving throughput than prior state-of-the-art systems like FasterTransformer and Orca at the same latency [2]. Across requests, providers reuse cached prefixes, sold to developers as prompt caching with large input-token discounts for the reused portion.

Example

An agent platform team notices their coding agent's bill is dominated by input tokens: every turn resends the same 30,000-token prefix of system prompt, tool definitions, and repo conventions, followed by the growing conversation. Because the prefix is byte-identical across turns, the provider's cache serves it at a fraction of the normal input price, but only if it stays identical. An engineer then adds a current timestamp near the top of the system prompt, and cache hit rates collapse, tripling effective input cost overnight, since every request now has a different prefix from the first divergent token onward. The fix is ordering: static instructions and tools first, anything dynamic last. Latency improves alongside cost, because cached prefixes skip most of prefill.

What People Get Wrong

The common confusion is treating the KV cache as a response cache, as if the model were memoizing answers to repeated questions. It caches internal attention state, not outputs. A cache hit on your prompt prefix does not mean you get the same completion, and it does not require the same question, only the same leading bytes. This misunderstanding leads to real mistakes in both directions: teams expect identical answers from cached prompts and are surprised by variation, or they dismiss prompt caching as irrelevant because their questions differ, missing that their giant shared system prompt is exactly what the cache monetizes.

FAQ

Why does the KV cache use so much memory? Because it stores key and value vectors for every token at every layer. The per-token cost is fixed by architecture, so memory grows linearly with context length, and serving many long-context users in parallel multiplies it. This, more than model weights, is what limits concurrent long-context sessions on a GPU. It is also why cache management dominates serving performance: in vLLM's 2023 benchmarks on LLaMA models, paged cache management delivered up to 24x higher throughput than Hugging Face Transformers [3].

Is prompt caching the same thing as the KV cache? Prompt caching is the commercial surface of KV cache reuse: the provider keeps the computed cache for a prompt prefix and charges a reduced rate when a later request reuses it exactly. The underlying mechanism is the same cache that speeds up generation within a single request.

How do I make my agent cache-friendly? Keep the prompt prefix stable and byte-identical across turns: system prompt and tool definitions first, dynamic content like retrieved context and user messages last, and never inject timestamps or random IDs early in the prompt. Appending to the end of a conversation preserves the cached prefix; editing the beginning invalidates it.

Sources

1. vLLM project (official blog). "Existing LLM serving systems waste 60-80% of KV cache memory; PagedAttention reduces waste to near zero." https://vllm.ai/blog/2023-06-20-vllm. Accessed August 2026.
2. UC Berkeley (arXiv, SOSP 2023). "PagedAttention improves LLM serving throughput 2-4x over FasterTransformer and Orca at the same latency." https://arxiv.org/abs/2309.06180. Accessed August 2026.
3. vLLM project (official blog). "vLLM delivered up to 24x higher throughput than Hugging Face Transformers on LLaMA models." https://vllm.ai/blog/2023-06-20-vllm. Accessed August 2026.