Quantisation: trading bits-per-weight for memory you may not have needed

Quantisation in the model sense means storing a trained model's numbers at lower numerical precision than they were trained at — 8-bit integers instead of 32-bit floats, say. The Model compression article states the two payoffs plainly: low-precision parameters take up less space, and take less compute to do arithmetic with. The cost, in the article's words: quantised models run "at the cost of reduced model accuracy". (Everything below is summarised from the sources cited at the bottom; edited, not verified.)

Where it sits in the family

Model compression is the wider category — quantisation is one technique in it, alongside pruning (zeroing parameters to sparsify the matrix) and low-rank factorisation (replacing a weight matrix W with a product UVᵀ of thinner matrices). Two distinctions the sources make, because the terms get collapsed constantly:

• Compression ≠ distillation. Compression shrinks the model itself, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a new, smaller student to imitate the teacher's behaviour. A quantised 70B is still nominally 70B parameters; a distilled 8B is a different model. See Knowledge distillation.
• Quantise unevenly. The article notes some parameters can be quantised more aggressively than others — 8-bit for a less important one, 16-bit for a more important one — at the price of needing mixed-precision arithmetic at inference.

Quantisation is also not only post-training: the article records quantised numbers being used during training, with PyTorch's automatic mixed-precision doing autocasting, gradient scaling, and loss scaling.

What the LLM-serving stack actually did with it

The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting 2-bit to 8-bit quantised integer types, the usual float32/float16/bfloat16, and a 1.58-bit quantisation scheme. llama.cpp offers both ahead-of-time model quantisation and on-the-fly KV-cache quantisation — so the cache that KV caching says you must pay for is itself a quantisation target. The library's whole reason for existing was running on hardware without a GPU; quantisation is what made that arithmetically possible.

The "train big, then compress" result from the Deep Compression literature is the other number worth carrying: at the same compute budget, training a large model briefly and then compressing it heavily beat training a small lightly-compressed model. Deep Compression reported a 35× compression ratio on AlexNet. Both are pre-LLM results; treat the ratios as history, not expectation.

Where it bites (my reading, labelled as such)

None of the cited articles quantifies which accuracy is lost. What follows is inference from what they do say, not their claims:

• Accuracy is not one number. A quantised model can keep passing a coding benchmark while losing the long tail of rare-knowledge questions; a single headline pass rate will hide that. The wiki's own Benchmarking local models has measured rows at several quantisations of similar models — that page is the closest thing to ground truth on this wiki for "what does IQ2_XXS actually cost you", and it keeps the spread visible rather than trusting one run.
• The memory saving is the reliable part; the speed saving is conditional. "Less compute to do arithmetic with" assumes kernels that exploit the narrow type; on a CPU or unified-memory box the binding constraint is often memory bandwidth, where fewer bits moved is the whole game.
• Mixed-precision means mixed bugs. When different layers run in different widths, numerical behaviour varies by layer, which makes cross-model score comparisons fragile — the same reason the benchmark page warns against reading a single leaderboard delta as real.

Sources: Wikipedia, "Model compression" and "Llama.cpp", read 2026-09-08. Summary plus labelled inference — edited, not verified. The articles describe mechanisms and formats; nothing here is a measured claim about a specific quantised model. Related: KV caching, Benchmarking local models, Knowledge distillation — the other way to get a small model.