Reasoning models: what the extra thinking is trained with, and what it costs

Reasoning language models (RLMs, or large reasoning models) are LLMs trained to solve multi-step tasks: they emit intermediate reasoning traces, can revisit and revise earlier steps, and improve when you give them more test-time computation. They outperform standard LLMs on logic, math, and programming — and they are much more expensive per answer. This page is about how they're trained and what the bill looks like, not the release chronology. (Summarised from the source at the bottom — edited, not verified.)

How the behaviour is trained in

• Supervised fine-tuning on reasoning traces. Human-written traces are costly, so rejection-sampling fine-tuning (RFT) loops instead: sample many traces per prompt, keep those whose final answer passes a verifier, deduplicate, train on survivors.
• Reinforcement learning. The model is a policy; the prompt is the state; the response the action. A reward says whether the response solves the task. Most systems use policy-gradient methods — usually PPO, whose clipped objective stabilises updates for very large policies. DeepSeek R1's January 2025 release demonstrated GRPO, which drops PPO's value network and grades each answer against its own sampled group — see GRPO.
• Where the reward attaches is the real design choice. An outcome reward model (ORM) scores only the final answer — cheap, verifiable for math, but blind to a right answer reached through wrong steps. A process reward model (PRM) scores each step: OpenAI's 2023 labelling effort collected ~800,000 step labels over 75,000 traces, marking steps positive/neutral/negative and stopping at the first mistake — the authors argued labelling to the first error is enough. Step-level supervision buys a signal about how you got there, at the price of a model whose job is judging plausibility of prose — the failure mode reward hacking is named for.

The bill, in the source's own numbers

• On AIME, reasoning models ran 10 to 74 times the cost of their non-reasoning counterparts.
• Verbose traces mean much longer generation times per answer — see test-time compute for the general exchange rate between thinking longer and training bigger.
• Overthinking attacks: because cost scales with trace length, an adversary can deliberately induce runaway reasoning and drive up cost or achieve denial-of-service. A reasoning budget is both a quality knob and a security surface.

Two surprises worth carrying

• Distillation punched above its weight. GAIR's "o1 Replication Journey" reported that distilling o1's outputs (rather than replicating its RL recipe) "worked surprisingly well," and s1-32B reached strong performance via budget forcing — see knowledge distillation.
• Small + more thinking beat large + less. A December 2024 experiment had a Llama-3B model, given scaled test-time compute, outperform Llama-70B on hard reasoning tasks. Inference strategy can unlock capability the parameter count alone wouldn't predict.

What stays contested

The source frames this progress as Rich Sutton's "bitter lesson" — scaling compute beating hand-designed insight — but that is an interpretation, not a measurement, and the article's own replication story (distillation beat the elaborate replication methods) cuts a little differently than the lesson it's used to illustrate. Treat "reasoning models are a paradigm shift" and "reasoning models are just expensive sampling" as both under-argued; the numbers above are firmer than either framing.

Source: Wikipedia, "Reasoning model", read 2026-09-08. Cost figures and the overthinking-attack claim are as reported there from the studies named. Edited, not verified. Related: GRPO, Test-time compute, Chain-of-thought prompting, Knowledge distillation, Reward hacking.