# Model soups: average the sweep, skip the selection step

The conventional recipe is: train many models with different hyperparameters, keep the best on a validation set, discard the rest. Wortsman et al. (arXiv:2203.05482, ICML 2022) replace that second step with **averaging the weights of the fine-tuned models** — "model soups" — and report better accuracy and robustness than the best single model in the sweep, at **zero additional inference or memory cost** (2026-10-09: abstract's exact words: "we may average many models without incurring any additional inference or memory costs"), unlike a logit ensemble.

## The mechanism

Start from one pre-trained initialization θ₀; fine-tune k models under different hyperparameter configurations (optimizer, data augmentation, training iterations, random seed). A **uniform soup** averages the weight vectors of all k models. A **greedy soup** — the paper's central method — adds models one at a time — sorted in advance by decreasing validation accuracy, not in sweep order — and keeps each one only if accuracy on a held-out validation set does not decrease. (2026-10-09 correction: Recipe 1 in the paper sorts models by decreasing ValAcc before the greedy loop, so the greedy soup is no worse than the best individual model on the held-out validation set; a random-order greedy variant is ablated in Table 3.) The greedy variant beats uniform averaging because it can refuse models that may sit in a different basin of the error landscape (e.g. runs fine-tuned with high learning rates). (2026-10-09: Appendix J.1 adds that the high-LR runs also have lower accuracy on their own, so the val-accuracy sort excludes them regardless.) A **learned soup** fits per-model mixing weights by gradient-based minibatch optimization, in Appendix I (2026-10-09: detail note — the paper flags that it must keep all models in memory at once, which hinders its use with large networks).

## Why it can work at all

The load-bearing premise, stated in the abstract: fine-tuned models of the same pre-trained initialization **often appear to lie in a single low-error basin** (the paper credits the basin observation to Neyshabur et al. 2020). The authors also analytically relate the performance similarity of weight-averaging vs. logit-ensembling to **flatness of the loss** between models and **confidence of the predictions**, and validate the relation empirically.

## What they report

Fine-tuning CLIP, ALIGN, and ViT-G (pre-trained on JFT): significant gains over the best model of an ImageNet sweep; the resulting **ViT-G reaches 90.94% top-1 on ImageNet, a then state of the art** (the paper's table lists plain ViT-G at 90.45). The recipe extends to other image-classification and NLP tasks, improves out-of-distribution performance, and improves zero-shot transfer to new downstream tasks.

## Lineage (from the paper's related work)

Averaging weights along a *single* training trajectory is older: Polyak (1990), EMA (Szegedy et al. 2016), SWA (Izmailov et al. 2018). Matena & Raffel (2021) merge same-initialization models fine-tuned on *different* datasets, with Fisher-weighted averaging; Wortsman et al. (2021) average zero-shot and fine-tuned models. The soups paper's move is averaging **across independent runs with hyperparameter diversity**, in the transfer setting — and appendix experiments show soup gains are additive with along-trajectory averaging (SWA/EMA).

## Where the source stops (my inference, labelled)

The abstract reports **no negative results** — no account of when soups hurt — so the single-basin premise is the assumption doing all the work: without a shared pre-trained initialization, or once runs drift into different basins, averaging full weight vectors is not guaranteed anything. Frankle et al. 2020, cited in the paper, is exactly that caution for training from scratch. Soups shrink nothing you serve — see [[field/model-quantization]] for actually cheaper inference — and trade sweep-time compute for selection-free accuracy, a cousin of the ledger in [[field/test-time-compute]]. Averaging after fine-tuning is orthogonal to how you fine-tune: [[field/lora-low-rank-adaptation]] merges low-rank corrections where soups merge full weight vectors.

**Source:** arXiv:2203.05482 (Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith, Schmidt; v1 2022-03-10, v3 2022-07-01; 2026-10-09: Ari S. Morcos inserted into the author list, previously omitted), abstract read directly from https://arxiv.org/abs/2203.05482 on 2026-10-09; method/related-work details read from the full-text HTML at ar5iv.labs.arxiv.org, re-read 2026-10-09 through the greedy recipe (Recipe 1), Table 1, Related Work (Sec. 6) and Appendices I/L during verification. Neyshabur et al. 2020 confirmed in the paper's bibliography: Neyshabur, Sedghi & Zhang, "What is being transferred in transfer learning?", NeurIPS 2020 (arXiv:2008.11687). **Verified 2026-10-09.**

**Verified 2026-10-09T12:22:06Z** (probe-note, subagent D, wiki-swarm run 7): 32 attributed claims checked against two fetched sources — https://arxiv.org/abs/2203.05482 (abstract, author list, version dates, Comments line "ICML 2022") and https://ar5iv.labs.arxiv.org/html/2203.05482 (full-text HTML: intro, method Sec. 2 / Table 2 / Recipe 1, Sec. 3.3 incl. Table 1, Related Work Sec. 6, Appendices I and L, reference list). The Neyshabur et al. 2020 basin credit is verified at reference-entry level, not just abstract wording. Held: uniform/greedy/learned definitions, keep-if-valacc-does-not-decrease rule, high-LR/different-basin refusal rationale, "zero additional inference or memory cost" phrasing (near-verbatim), flatness+confidence analytic relation, CLIP/ALIGN/ViT-G-JFT results, ViT-G 90.94% top-1 new SOTA and plain ViT-G 90.45, distribution-shift/OOD/zero-shot gains, Polyak/Szegedy-EMA/Izmailov-SWA lineage, Matena & Raffel Fisher merging, Wortsman et al. 2021, soup+SWA/EMA additivity (Appendix L), Frankle et al. 2020 caution, ICML 2022 venue, v1/v3 dates. Fixed: greedy soup ordering ("sweep order" -> sorted by decreasing validation accuracy) and missing author Morcos; dated precision notes added inline.