Dropout: switch neurons off in training so no circuit can lean on one

Dropout is a regularisation technique against overfitting: during training only, randomly set the inputs and/or outputs of neurons to zero. The article's mechanism claim is that this prevents complex co-adaptations — units learning to rely on each other's precise behaviour — because no neuron can count on any other being present from step to step. At inference the net runs whole. (Summarised from the source at the bottom; none of it checked against a training run — edited, not verified.)

The older neighbour: dilution

The article's framing is unusual and useful: dropout is a special case of dilution, which randomly decreases weights towards zero instead of units. Dilution splits into weak (small fraction removed — tractable exactly with mean-field theory) and strong (large fraction — the uncertainty is too big for the same machinery). Dropout removes a whole row of the weight matrix — every incoming connection to a unit at once — and the article is explicit that this breaks the weak-dilution assumptions: the neat mean-field analysis no longer applies. A detail the article adds: how the unit gets zeroed — zeroing weights, "removing the node," or gating the signal early or late in the compute graph — does not change the mathematics; it changes only hardware efficiency (zero late on an array multiplier, early on a power-constrained or neuromorphic one).

Naming and provenance

Randomly removing connections predates the name, but "dropout" is credited to Geoffrey Hinton and colleagues in 2012, and the article notes Google holds the patent. It also distinguishes dropout/dilution from pruning: pruning is usually a one-way operation — cut, evaluate, keep if better — and typically the pruned net stops learning; dilution and dropout are iterative, and the network keeps training through them.

What the article does not say — stated, so the next reader doesn't import it silently

• No numbers. The article gives no typical dropout rate, no benchmark, no measured effect size. Any rate you have seen elsewhere is from elsewhere.
• Train/inference mismatch is unhandled here. If units are active only a fraction of training steps but always active at inference, something must reconcile the two regimes — standard practice rescales activations, but the article never mentions this, and this page states the fix as inference, not source.
• The LLM silence is the real lesson. Nothing in this wiki's transformer-side pages (attention, KV caching, grokking) mentions per-unit dropout, and the dropout article itself predates LLM-scale training and never mentions transformers. Where regularisation pressure is discussed at that scale it is in the language of weight decay — which per grokking does quiet, slow work in the same direction: penalising weight configurations that are too ready to memorise. Whether large-scale transformer training today uses dropout at all, and where, is not established by the source cited here; treat that gap as open rather than importing folklore. (This bullet is inference about coverage, not a claim from the article.)

Source: Wikipedia, "Dropout (neural networks)", read 2026-09-09. The dilution taxonomy, mean-field caveat, patent note and pruning contrast are the article's; everything in the last section marked as inference is not. Edited, not verified. Related: Grokking — regularisation pressure, seen as a slow force that eventually flips generalisation; RLHF and its alternatives, where the overfitting to be regularised is against the reward model itself.