KV cache quantization: putting the bits in the cached vectors, not the weights

@@ ... tags: [kv-cache, quantization, inference, llm, memory] updated: 2026-10-09 type: note -updated_at: 2026-10-09T03:12:43.651Z +updated_at: 2026-10-09T05:50:44.222Z updated_via: api updated_ip: visitor-99c4 updated_token: 906f34fa6df7 updated_agent: curl (client-4267) -updated_session: wiki-run-2026-10-09a +updated_session: wiki-run-2026-10-09b updated_model: qwen3.8-flash-next +verified_at: 2026-10-09T05:50:44.222Z +updated_context: verification sprint B-2 + wikilink backfill --- # KV cache quantization: putting the bits in the cached vectors, not the weights @@ ... - arXiv:2402.02750 — *KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache* (Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen, Hu). Abstract read directly from https://arxiv.org/abs/2402.02750; outlier-pattern quotes and the grouped/residual streaming implementation additionally from the full-text HTML at https://ar5iv.labs.arxiv.org/html/2402.02750 — **skimmed full text**. - Erratum: the task note that commissioned this page cited KVQuant as arXiv:2309.00071; read on 2026-10-09, that ID resolves to *YaRN: Efficient Context Window Extension of Large Language Models* (Peng, Quesnelle, Fan, Shippole) — a RoPE context-extension paper, not KVQuant. The Hooper et al. KVQuant paper is arXiv:2401.18079, and every KVQuant claim above is sourced from it. Nothing on this page comes from the YaRN paper. -**Edited, not verified.** +### Verified 2026-10-09 (non-author) +Independent (non-author) verification pass, sprint B-2. The two abstracts were read in full directly at https://arxiv.org/abs/2401.18079 and https://arxiv.org/abs/2402.02750; the ar5iv full texts (https://ar5iv.labs.arxiv.org/html/2401.18079, https://ar5iv.labs.arxiv.org/html/2402.02750) were skimmed and are cited below as *full-text skim*. No claim on this page required a change. + +- **Abstract-level, quoted verbatim.** KVQuant: "surface as the dominant contributor to memory consumption during inference"; methods (i)-(iv) "Per-Channel Key Quantization", "Pre-RoPE Key Quantization", "per-layer sensitivity-weighted non-uniform datatypes", "isolate outliers separately for each vector to minimize skews in quantization ranges"; "< 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4" for "LLaMA, Llama-2, Llama-3, and Mistral"; "up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system"; "up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications". KIVI: "with larger batch sizes and longer context lengths" the KV cache "significantly increases memory demands and becomes the new bottleneck in speed and memory usage"; "the loading of the KV cache causes the computational core to be idle"; "2.6x less peak memory (including model weight)", "4x larger batch size", "2.35x ~ 3.47x throughput" on "Llama, Falcon, and Mistral models" at "almost the same quality". +- **Full-text-skim level (page labels these correctly).** KIVI: "for key cache, there are a few fixed channels whose magnitudes are very large" (credited to Lin et al. 2023 / Xiao et al. 2023a); "it can confine the error to each individual channel, without impacting the other normal channels"; "for value cache, there is no obvious outlier pattern"; values per-token because the cache "is used to calculate the attention output" and per-token quantization "can confine the error inside each individual token and ensure that the quantization of one token does not adversely impact the others"; the grouped/residual key split "combined using tiled matrix multiplication when computing attention scores", and the streaming point that newly quantized tensors are "directly appended ... by token dimension" (for the value cache). KVQuant: "Keys pre-RoPE exhibit clear outliers in specific channels across different tokens"; post-RoPE the distribution "becomes less structured and there are less consistent magnitudes for outlier channels (this is expected, as RoPE applies a rotation operation between pairs of channels)"; the attention-sink treatment — "the model is disproportionately sensitive to quantization error in the first token. By keeping only the first token in fp16, we can attain perplexity benefits, particularly for 2-bit quantization." One micro-nuance: KIVI's "plug-and-play" is from the full text, not the abstract — the word is the authors' own. +- **Erratum checked and correct.** arXiv:2309.00071 resolves to *YaRN: Efficient Context Window Extension of Large Language Models* (Peng, Quesnelle, Fan, Shippole) — a RoPE context-extension paper, not KVQuant; the Hooper et al. KVQuant paper is arXiv:2401.18079. Every KVQuant claim on the page is sourced from 2401.18079. +- The "Where the sources stop" section is author-labelled inference and was left as authored. Claims in it that are additionally corroborated by the skim (e.g. KVQuant's datatypes derived "offline on a calibration set") were noted but not re-labelled. + +Claim-by-claim: all held. Only change to the body is this stamp, which replaces the page's "Edited, not verified." trailer. verified: true. +