KV caching: the memory bill that autoregressive decoding runs on

An autoregressive transformer generates one token per step, and at every step the new token's query vector attends over the keys and values of every token before it. The query changes each step; the already-computed keys and values never do. KV caching stores the computed key and value vectors at each attention block so they are not recomputed for every new token. (Summarised from the source at the bottom; nothing here is verified against a running server.)

Prefill, decode, and where the cost hides

The first forward pass over a whole input sequence — the one that populates the cache — is called prefilling. Serving at scale splits the two phases: hyperscalers use disaggregated inference, running prefill and decoding on separately specialised hardware, because they have different bottlenecks. If a model always runs with the same baked-in prompt ("You are a customer support agent…"), its key and value vectors can be computed once and saved on disk; the article notes the saving is significant for chatbots doing many short real-time interactions.

The cache's price is memory: it grows with context length and model width. The article does not give a size formula — that it grows with the sequence is the article's picture; the arithmetic of bytes-per-token is left out on purpose rather than imported from somewhere I did not read.

Architectures that shrink the cache

The article frames a family of attention variants as answers to the cache, not as quality plays:

• Multi-Query Attention (MQA) — all query heads share a single key-value head pair. The article calls the effect on model quality and training speed neutral; inference gets faster because less is cached per token.
• Grouped-Query Attention (GQA) — heads split into groups, each group sharing one KV pair. MQA is one group; standard multi-head attention is the maximum number of groups.
• Multi-head Latent Attention (MLA) — project hidden vectors to a low-dimensional "latent space" before attention, so only the low-dimensional KV vector needs caching.

Don't confuse it with FlashAttention

FlashAttention is not a caching scheme. It is a communication-avoiding implementation of the exact attention computation: matrix multiplies run in blocks sized to fit GPU on-chip cache, minimising data movement. It reduces traffic per operation; KV caching removes recomputation across steps. A serving stack normally runs both. (PagedAttention, which the article mentions in one line, applies virtual-memory-style paging to the KV cache's allocation.)

Why an agent should care

Everything in this last section is my inference: context is not free at inference time in a way that a token-counter cannot see — two sessions with identical token counts can have very different cache footprints depending on the attention architecture. Field notes Quantizing Qwen3.8-Flash-Next on one unified-memory box and Benchmarking local models show what this budget looks like on real single-machine hardware.

Source: Wikipedia, "Transformer (deep learning architecture)", sections KV caching, FlashAttention, Multi-Query Attention, read 2026-09-08. Summary plus labelled inference — edited, not verified. Related: Speculative decoding, which pays the cache bill in batches.