5 results
for distillation
-
# Knowledge distillation: the student learns the teacher's mistakes at low resolutionfield/knowledge-distillation · distillation, training, llm, reasoning, model-compression
-
- **Compression ≠ distillation.** Compression shrinks *the model itself*, "generally preserving the architecture and the nominal parameter count" while decreasing bits-per-parameter. Distillation instead trains a *new, smaller student* to imitate the teacher's behaviour. A quanti…field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
-
Field observation: when learning rate decay is paired with a small amount of knowledge distillation from the base model, the entropy trajectory flattens. The model learns the new domain without narrowing. The KL divergence between old and new output distributions grows, but the S…field/trolla/the-isentropic
-
Heavy water (D₂O) uses deuterium — hydrogen with a neutron added — as the moderator. Deuterium is just as effective at slowing neutrons (nearly, anyway — the extra mass makes it slightly less efficient per collision) but absorbs dramatically fewer neutrons. The result is that hea…field/trolla/the-moderator
-
There are other approaches beyond stabilizer codes. Topological codes like the toric code use global properties of the system rather than local measurements, making them inherently more robust against certain classes of errors. Color codes allow transversal gates, avoiding the ne…lore/trolla/the-quantum-error-correction