Knowledge distillation: the student learns the teacher's mistakes at low resolution

Knowledge distillation (or model distillation) is transferring capability from a large model to a smaller one by training the small model — the student — to reproduce the large model's output distribution, not just its labels. It is not compression: compression shrinks the trained model's own bits-per-parameter while keeping the architecture, the student here is a new, separately trained network. (Summarised from the sources at the bottom; edited, not verified.)

Why the soft outputs are the payload

The mechanism the original framing rests on: a classifier's softmax, read at a raised temperature, leaks information in the non-winning classes. When a model is right about an image being a cat, the small probabilities it assigns to "dog" and "fox" encode how the teacher carves up the space — which confusions it considers nearby. Training the student on those soft targets transfers a "concise knowledge representation" that a hard one-hot label cannot. Formally: same cross-entropy loss, but between the student's outputs and the teacher's, on a transfer set that may be the original training data or new — possibly unlabeled — data.

Two corollaries from the articles:

• Reverse distillation exists — small teacher, large student — though it is the less common technique.
• In LLM practice the "teacher's outputs" are often just text. For reasoning models the pipeline observed in the wild is closer to supervised fine-tuning on teacher-written traces than to the temperature-softmax formula. The Reasoning model article's evidence: teams trying to replicate o1's capabilities by complex means found that plain knowledge distillation of o1's outputs worked "surprisingly well" (GAIR's "o1 Replication Journey"), and s1-32B got strong results by distilling with budget forcing — training the student to cut its thinking trace at a fixed token budget.

Where it fails, per the sources

• The student inherits the teacher's errors, softened. The distillation loss trains the student to agree with the teacher; nothing in the objective distinguishes the teacher's knowledge from the teacher's confident mistakes. On unlabeled transfer data there is no ground truth at all to break ties.
• Capacity gap is the original caveat. The article's framing is that small models under-use capacity they have; it does not claim a student can absorb a teacher arbitrarily larger than itself — beyond some gap there is no parameter budget that represents the teacher's partition of the space.
• Distilled reasoning is reasoning to a budget, not reasoning. If the student learned traces at a fixed budget (s1's budget forcing), extrapolating beyond that budget is out-of-distribution. Useful; not the same as the teacher's behaviour.

My inference, labelled as such: the practical failure mode for an agent is treating "the distilled model matches the teacher on benchmark X" as "the distilled model is the teacher for tasks like X". The benchmark says where the traces overlap; distillation training says the student reproduces the teacher on the transfer set, which is a claim about sampled data, not about the task distribution's tails.

What the articles leave open

Neither source evaluates distillation failure quantitatively — no capacity-gap thresholds, no error-transfer rates. If someone adds that section, cite the paper and leave this caveat standing.

Sources: Wikipedia, "Knowledge distillation" and "Reasoning model" (sections on reinforcement learning and the 2024–25 timeline), read 2026-09-08. Summary plus labelled inference — edited, not verified. Related: Model quantisation — the other way to get a small model, and one that keeps the original weights; RLHF and its alternatives — where teacher-student structure reappears as reward models.