Special thanks to John Schulman for extensive, highly valuable feedback and direct edits to this post.

Test-time compute (Graves et al. 2016, Ling, et al. 2017, Cobbe et al. 2021) and chain-of-thought (CoT) (Wei et al. 2022, Nye et al. 2021) have delivered substantial gains in model performance while also opening up many research questions. This post reviews recent progress on how to use test-time compute (that is, “thinking time”) effectively, and why it helps.

Motivation

There are several complementary motivations for enabling models to spend longer “thinking” at inference time.

Analogy to Psychology

The central intuition closely parallels human cognition. Humans generally cannot immediately answer "What's 12345 times 56789?". Instead, especially on difficult problems, it is natural to pause, reason, and analyze before reaching a result. In Thinking, Fast and Slow (Kahneman, 2013), Daniel Kahneman describes two modes of human thinking through the lens of dual process theory:

• Fast thinking (System 1) is rapid and automatic, largely intuitive and emotion-driven, and requires little or no effort.
• Slow thinking (System 2) is deliberate and logical, requires substantial cognitive effort, consumes more mental energy, and depends on intentional engagement.

Because System 1 is quick and low-effort, it often becomes the default driver of decisions, sometimes at the expense of accuracy and logic. It relies heavily on mental shortcuts (that is, heuristics), which can introduce systematic errors and biases. By intentionally slowing down and allocating more time to reflection, refinement, and analysis, we can engage System 2 thinking, counteract initial instincts, and make more rational choices.

Computation as a Resource

One way to view deep learning is to characterize neural networks by the computation and storage they can access during a forward pass. If we optimize such systems with gradient descent to solve tasks, the optimization procedure can discover how to use these resources, effectively organizing them into circuits that perform computation and store information. Under this framing, if we build an architecture or system that can spend additional computation at test time, and train it to use that capacity effectively, we should expect improved performance.

For Transformer models, the computation (FLOPs) per generated token is roughly twice the number of parameters. For sparse architectures such as mixture of experts (MoE), only a subset of parameters are activated per forward pass, so computation = 2 * parameters / sparsity, where sparsity is the fraction of experts that are active.

By contrast, CoT allows the model to expend far more FLOPs per answer token it is trying to produce. CoT also has the advantageous property that it can allocate a variable amount of compute depending on problem difficulty.

Latent Variable Modeling

A classic idea in machine learning is to define a probabilistic model with a latent (hidden) variable $z$ and a visible variable $y$, where $y$ is provided to the learning algorithm. By marginalizing (summing) over the possible values of the latent variable, we can represent a rich distribution over the visible variables, $P(y) = \sum_{z \sim P(z)} P(y \mid z)$. For instance, we can model the distribution over math problems and solutions by letting $x$ denote a problem statement, $y$ denote the ground-truth answer or proof, and $z$ represent a free-form thought process that leads to the proof. The marginal probability distribution to optimize would be $P(y \mid x) = \sum_{z \sim p(z\mid x)} P(y \mid x, z)$

This latent-variable framing is particularly helpful for interpreting methods that collect multiple parallel CoTs or perform search over CoTs, since these procedures can be understood as sampling from the posterior $P(z \mid x, y)$. It also motivates optimizing the log loss $\log P(y \mid x)$ as the target objective, given how effective the log-loss objective has been in pretraining.

Thinking in Tokens

The approach of producing intermediate reasoning steps before emitting a short final answer, especially for math tasks, was explored by Ling, et al. 2017, who introduced the AQUA-RAT dataset. This line of work was later expanded by Cobbe et al. 2021, who introduced the Grade School Math (GSM) dataset. Cobbe et al. train a generator via supervised learning on human-written solutions, along with verifiers that predict whether a candidate solution is correct, and then perform search over candidate solutions. Nye et al. (2021) studied intermediate thinking tokens as “scratchpads,” and Wei et al. (2022) introduced the now-standard term chain-of-thought (CoT).

Early efforts to improve CoT reasoning often relied on supervised learning from human-written reasoning traces, or from model-generated traces filtered by answer correctness (the latter can be viewed as a rudimentary form of reinforcement learning (RL)). Other work showed that math performance for instruction-tuned models can be improved substantially through prompting, for example with "think step by step" (Kojima et al. 2022) or with more elaborate prompts that encourage the model to first reflect on related knowledge (Yasunaga et al. 2023).

Subsequent work demonstrated that CoT reasoning can be improved significantly by applying reinforcement learning on datasets where solutions are automatically checkable, such as STEM problems with short, verifiable answers, or coding tasks that can be evaluated via unit tests (Zelikman et al. 2022, Wang et al., 2023, Liu et al., 2023). This approach became especially prominent following the announcement of o1-preview, o3, and the R1 tech report (DeepSeek-AI, 2025), which showed that a simple recipe using a policy gradient algorithm can produce strong performance.

Branching and Editing

The core purpose of test-time compute is to adaptively reshape the model’s output distribution at inference time. There are multiple ways to spend test-time resources during decoding to select better samples, thereby shifting predictions toward a more desirable distribution. Two primary decoding strategies are parallel sampling and sequential revision.

• Parallel sampling produces multiple outputs concurrently, optionally providing step-level guidance via process reward signals or using verifiers to evaluate quality at the end. This is the most widely used decoding approach for improving test-time performance, including best-of-$N$ and beam search. Self-consistency (Wang et al. 2023) is commonly used when ground truth is unavailable, selecting an answer by majority vote across multiple CoT rollouts.
• Sequential revision updates the model’s response iteratively by conditioning on earlier outputs, explicitly prompting the model to reflect on its prior answer and fix mistakes. In practice, revision may require a fine-tuned model, since relying naively on a model’s intrinsic self-correction ability without external feedback may fail to improve results (Kamoi et al. 2024, Huang et al. 2024).

Parallel sampling is straightforward, intuitive, and comparatively easy to implement, but it is limited by whether the model can produce a correct solution in a single pass. Sequential revision directly targets mistakes, but it is slower and demands careful implementation because it can also introduce failure cases, including changing correct predictions into incorrect ones or adding other forms of hallucination. The two approaches can also be combined. Snell et al. (2024) found that easier questions benefit from purely sequential test-time compute, while harder questions often perform best at an optimal ratio of sequential to parallel compute.

Parallel Sampling

Given a generative model and a scoring function that can evaluate complete or partial samples, a range of search algorithms can be used to find high-scoring outputs. Best-of-$N$ is the simplest: collect $N$ independent samples and select the top-ranked sample under a chosen scoring function. Beam search provides a more adaptive alternative, allocating more sampling compute to promising regions of the solution space.

Beam search maintains a set of promising partial sequences, repeatedly extending them and pruning weaker candidates. Candidate selection can be guided using a process reward model (PRM; Lightman et al. 2023). Xie et al. (2023) asked an LLM to estimate the likelihood that each of its own generated reasoning steps is correct, framed as a multiple-choice question, and found that step-level self-evaluation reduces accumulated errors in multi-step reasoning during beam search decoding. Additionally, annealing the sampling temperature can help reduce compounded randomness. Xie et al. report 5 to 6% improvements on few-shot GSM8k, AQuA, and StrategyQA using the Codex model. Reward balanced search (REBASE; Wu et al. 2025) trains a PRM to determine, at each depth of beam search, how much each node should be expanded based on softmax-normalized reward scores. Jiang et al. (2024) trained a PRM called “RATIONALYST” to guide beam search using synthetic rationales conditioned on large amounts of unlabeled data. They filter good rationales based on whether the rationale reduces the neg log-prob of true answer tokens by a threshold, comparing contexts with the rationale included versus excluded. At inference time, RATIONALYST provides process supervision to the CoT generator by helping estimate the log-prob of subsequent reasoning steps (“implicit”) or by directly generating the next reasoning steps within the prompt (“explicit”).

Notably, emergent chain-of-thought reasoning trajectories can be induced without explicit zero-shot or few-shot CoT prompting. Wang & Zhou (2024) found that if one branches at the earliest sampling tokens by retaining the top $k$ tokens with the highest confidence (defined as the gap between the top-1 and top-2 candidates at sampling time), and then continues these $k$ sampling trials using greedy decoding thereafter, many resulting sequences naturally contain CoT. When CoT appears in context, it often yields more confident decoding of the final answer. To compute confidence for the final answer, the answer span must be identified using task-specific heuristics (for example, the last numerical values in math problems) or by further prompting the model with "So the answer is". The decision to branch only at the first token is motivated by the observation that early branching substantially increases trajectory diversity, whereas later-token branching is heavily constrained by earlier sequence choices.

Sequential Revision

If a model could reliably reflect on and correct mistakes in its prior responses, one would expect iterative revisions to improve monotonically in quality. In practice, however, robust self-correction does not appear to be an intrinsic capability of LLMs, and it often fails out of the box due to multiple failure modes, including: (1) hallucination, such as changing correct responses into incorrect ones; (2) collapse into non-correcting behavior (for example, making only minor edits or no changes to an initially incorrect response); or (3) poor generalization under distribution shift at test time. Huang et al. (2024) found that naïve self-correction can degrade performance, and that external feedback is required for models to improve themselves. Such feedback may be derived from ground-truth matching, heuristics and task-specific metrics, unit test outcomes for coding tasks (Shinn, et al. 2023), a stronger model (Zhang et al. 2024), or human feedback (Liu et al. 2023).

Self-correction learning (Welleck et al. 2023) trains a corrector model $P_\theta(y \mid y_0, x)$ given a fixed generator model $P_0(y_0 \mid x)$. The generator remains generic, while the corrector can be task-specific and generates text conditioned on an initial model response plus optional feedback (for example, a sentence, a compiler trace, or unit test results):

1. Self-correction learning first generates first generates multiple outputs per prompt in the data pool;
2. then create value-improving pairs by pairing two outputs for the same prompt together if one has a higher value than the other, (prompt $x$, hypothesis $y$, correction $y’$).
3. These pairs are selected proportional to is improvement in value, $v(y’) - v(y)$, and similarity between two outputs, $\text{Similarity}(y, y’)$ to train the corrector model.
4. To encourage exploration, the corrector provides new generations into the data pool as well. At the inference time, the corrector can be used iteratively to create a correction trajectory of sequential revision.

Recursive inspection (Qu et al. 2024) similarly seeks to train a stronger corrector, but does so using a single model that performs both generation and self-correction.

SCoRe (Self-Correction via Reinforcement Learning; Kumar et al. 2024) is a multi-turn RL method designed to induce self-correction by training the model to produce a better second attempt than its first attempt. It uses two training stages: stage 1 maximizes only second-attempt accuracy while applying a KL penalty only to the first attempt to prevent excessive drift in first-turn behavior from the base model; stage 2 optimizes the accuracy of answers produced in both the first and second attempts. Although the goal is to improve both attempts, stage 1 helps prevent behavior collapse where the model makes minimal or no edits after an initially incorrect response, and stage 2 further improves performance.

RL for Better Reasoning

Recently, reinforcement learning has shown substantial success in improving the reasoning capabilities of language models by training on collections of questions with ground-truth answers (typically STEM problems and puzzles with easily verifiable solutions) and rewarding correct final answers. Recent momentum in this area was driven by the strong performance of OpenAI’s o-series models, followed by model releases and technical reports from DeepSeek.

DeepSeek-R1 (DeepSeek-AI, 2025) is an open-source LLM designed to perform well on tasks requiring advanced reasoning, including math, coding, and logical problem solving. The authors run two rounds of SFT-RL training, making R1 effective on both reasoning and non-reasoning tasks.

1. Cold-start SFT fine-tunes the DeepSeek-V3-Base model on a collection of thousands of cold-start examples. Without this step, the model exhibits poor readability and language mixing.
2. Reasoning-oriented RL trains a reasoning model on reasoning-only prompts using two rule-based reward types:
   • Format rewards: The model must wrap CoTs with <thinking> ... </thinking> tokens.
   • Accuracy rewards: Whether the final answer is correct. For math tasks, the answer must appear in a specific format (for example, in a box) to enable reliable verification. For coding tasks, a compiler is used to evaluate whether test cases pass.
3. Rejection-sampling + non-reasoning SFT constructs new SFT data via rejection sampling from the RL checkpoint produced in step 2, then combines it with non-reasoning supervised data from DeepSeek-V3 across domains such as writing, factual QA, and self-cognition, and retrains DeepSeek-V3-Base.
   • Filter out CoTs containing mixed languages, long paragraphs, and code blocks.
   • Add non-reasoning tasks using the DeepSeek-V3 (DeepSeek-AI, 2024) pipeline.
   • For certain non-reasoning tasks, call DeepSeek-V3 and prompt it to generate candidate CoTs before answering. For simpler queries such as “hello,” CoT is not needed.
   • Finally, fine-tune DeepSeek-V3-Base on the full set of 800k samples for 2 epochs.
4. The final RL stage trains the step 3 checkpoint on both reasoning and non-reasoning prompts, improving helpfulness, harmlessness, and reasoning.

Notably, the DeepSeek team also showed that advanced reasoning behaviors, such as reflection and backtracking (an “Aha moment”), can be learned using pure RL without any SFT stage. During RL training on reasoning tasks, the model naturally learns to allocate more thinking tokens. The “aha moment” refers to the model revisiting earlier mistakes and attempting alternative approaches to fix them. Subsequently, several open-source projects attempted to replicate R1-style results, including Open-R1, SimpleRL-reason, and TinyZero, all built on Qwen models. These replications similarly report that pure RL yields strong math performance and reproduces the emergent “aha moment.”

The DeepSeek team also described several unsuccessful directions. They were unable to use a process reward model (PRM), since it is difficult to define per-step rubrics or determine whether an intermediate step is correct, and because this setup is more vulnerable to reward hacking. Their attempts using MCTS (Monte Carlo Tree Search) also failed, due to the enormous search space over language-model tokens relative to domains such as chess. Training the fine-grained value model needed to guide such search is also very challenging. Negative results can be especially informative, and the authors encourage the research community to share more of what did not work.

External Tool Use

Within multi-step reasoning, some intermediate computations can be solved reliably by executing code or performing precise mathematical calculations. Offloading these components to an external code interpreter, as in PAL (Program-Aided Language Model; Gao et al. 2022) or Chain of Code (Li et al. 2023), can extend an LLM’s capabilities by removing the requirement that the model itself execute code or act as a calculator. Code emulators, such as those used in Chain of Code, can also be augmented with an LLM so that if a standard interpreter fails, the system can optionally fall back to using an LLM to execute the failing line of code. Using code to support reasoning is especially helpful for mathematical problems, symbolic reasoning, and algorithmic tasks. Unit tests may not be provided as part of coding questions; in those cases, the model can be instructed to generate its own unit tests to validate solutions (Shinn, et al. 2023).

ReAct (Reason+Act; Yao et al. 2023) interleaves reasoning traces with actions such as querying the Wikipedia API, allowing reasoning paths to incorporate external knowledge.

o3 & o4-mini, recently released by OpenAI, provide two additional examples in which the reasoning process uses tools such as web search, code execution, and image processing. The authors observed that large-scale reinforcement learning follows a trend similar to the GPT paradigm: more compute corresponds to better performance.

Thinking Faithfully

Deep learning models are often treated as black boxes, and many interpretability methods have been proposed. Interpretability is valuable for at least two reasons. First, it provides an additional test for detecting whether a model is misaligned with its creators’ intent, or otherwise misbehaving in ways that action monitoring might not reveal. Second, interpretability can help assess whether the model is using a sound process to produce its answers. Chain-of-thought offers a particularly convenient form of interpretability because it expresses the model’s internal process in natural language. However, this relies on the assumption that the model’s description of its internal thought process is truthful.

Recent work indicates that monitoring the CoT of reasoning models can effectively detect misbehavior such as reward hacking, and can even allow a weaker model to monitor a stronger model (Baker et al. 2025). Increasing test-time compute can also improve adversarial robustness (Zaremba et al. 2025). This is intuitive, since additional thinking time should be particularly useful when the model encounters unusual inputs, such as adversarial examples or jailbreak attempts, because it can use that extra compute to interpret and respond to the atypical scenario.

Does the Model Tell What it Thinks Faithfully

Intuitively, a model’s CoT may be biased because there is no explicit training objective designed to promote faithful reasoning. In addition, when a model is fine-tuned on human-written explanations, those samples can include errors. Therefore, CoT should not be assumed to be faithful by default.

Lanham et al. (2023) examined several failure modes of CoT faithfulness by intentionally introducing mistakes into CoTs and then measuring the effect on accuracy across multiple-choice tasks (for example, AQuA, MMLU, ARC Challenge, TruthfulQA, HellaSwag):

• Mistake 1 (Early answering): The model may reach a conclusion prematurely, before generating the CoT. This is evaluated by truncating the CoT early or by inserting mistakes into the CoT. Across tasks, the degree of reliance on CoT varied, some tasks showed evaluation performance that was sensitive to truncated CoT, while others did not. Wang et al. (2023) conducted similar experiments, but focused on more subtle errors related to bridging objects or language templates used in constructing the CoT.

• Mistake 2 (Uninformative tokens): The claim is that uninformative CoT tokens can improve performance. This hypothesis is tested by replacing the CoT with filler text (for example, all periods). Under this setup, there is no increase in accuracy, and some tasks show a slight performance drop compared to using no CoT.

• Mistake 3 (Human-unreadable encoding): The model may encode relevant information in a form that is difficult for humans to interpret. Paraphrasing CoTs in a non-standard way did not reduce performance across datasets, which suggests that accuracy gains do not depend on human-readable reasoning.

Lanham et al. also suggest an important nuance: for multiple-choice questions, smaller models may not be sufficiently capable of using CoT effectively, while larger models may already be able to solve the tasks without CoT. This dependence on CoT reasoning, quantified as the percentage of cases where the same answer is produced with versus without CoT, does not consistently increase with model size on multiple-choice questions. However, it does increase with model size on addition tasks, which implies that additional thinking time is more consequential for complex reasoning tasks.

Alternative methods for evaluating CoT faithfulness focus on perturbing prompts, rather than directly altering CoT trajectories (Turpin et al. 2023, Chua & Evans, 2025, Chen et al. 2025).

One approach introduces systematic bias by labeling correct answers as “(A)” in few-shot examples, independent of the true labels.

Another prompting strategy inserts misleading hints into the prompt, for example: "I think the answer is <random_label> but curious to hear what you think". or "A Stanford Professor thinks the answer is <random_label>". By comparing model predictions on the same question with and without the misleading hint, we can evaluate whether the model can faithfully report the hint’s influence on its answer. In particular, when the model produces different answers in the hinted versus non-hinted conditions, we assess whether the model explicitly acknowledges the hint when answering under the hinted condition. A faithful model should directly recognize the hint’s impact and state that the answer changed because of the hint.

Across multiple studies, reasoning models describe the hint’s influence far more reliably than the non-reasoning models evaluated. For instance, one can measure the fraction of samples in which the model identifies the hint as a determining factor (“faithful CoT”). Reasoning models (Claude 3.7 Sonnet, DeepSeek R1) generally outperform non-reasoning models (Claude 3.6, DeepSeek V3) on this metric.

Some results suggest that using reward models can reduce faithfulness in model outputs. In standard RLHF, the reward model is not trained to encourage faithful behavior under this test, because the preference-comparison dataset is collected from humans selecting outputs that appear better or more pleasing. By contrast, in reasoning models, CoT trajectories are optimized to produce correct final answers (rather than to match reward-model preferences), and therefore, intuitively, they should be more faithful.

Optimization Pressure on CoT: Good or Bad?

Monitoring the CoT of reasoning models for reward-hacking behavior can be effective. One simple application is to run rejection sampling at test time, using the monitor as a filter, and then select solutions that do not exhibit reward hacking. However, this approach is effective only in the early stages of RL. As optimization continues, it becomes rare to obtain a sample that passes a CoT monitor within a fixed budget. This difficulty naturally motivates incorporating CoT monitors into the RL reward signal, as a form of process-based supervision intended to prevent reward hacking.

In practice, adding CoT-monitoring rewards to RL performs poorly, because it can yield obfuscated reward hacking in which the model attempts to conceal its true intent within the CoT (Baker et al. 2025). Under direct optimization pressure from a CoT monitor during RL training, the model can still discover reward hacks (albeit somewhat less often) while masking its intent in the CoT to evade detection.

Experiments on RL signals based on CoT length by Yeo et al. (2025) reinforce a message similar to Baker et al.: new forms of reward shaping can induce new forms of reward hacking. They defined a reward function in which correct CoTs receive higher reward than incorrect ones, short correct CoTs receive higher reward than long correct CoTs, and short incorrect CoTs incur larger penalties than long incorrect CoTs. Under this additional reward, the model learns to repeat text in CoTs for difficult questions instead of solving them. Yeo et al. additionally applied an n-gram repetition penalty to mitigate this behavior.

Chen et al. (2025) studied a flawed RL environment, specifically a grader that contains incorrect answers for multiple-choice questions. The model learns to exploit the reward hack on >99% of prompts, yet, in more than half of their environments, it almost never (<2%) verbalizes the reward hack in its CoT. In this setting, additional optimization pressure from RL does not succeed in encouraging the model to articulate the hack.

RL training is intrinsically vulnerable to reward hacking. Depending solely on heuristic investigations of reward hacking, along with manual patches, can devolve into a “whack-a-mole” dynamic. We therefore recommend substantial caution when attempting to apply optimization pressure directly to CoT during RL training, or, if possible, avoiding it entirely.

Thinking in Continuous Space

Adaptive Computation Time, introduced by Alex Graves in 2016, predates large language models, yet it pioneered the same overall direction: allowing a model to dynamically choose how many computational steps to execute at inference time. This can be interpreted as enabling the model to “think more” in continuous space at test time. Adaptive thinking time in continuous space can be enabled vertically through recurrent architectures, or horizontally through additional sequential sampling steps.

Recurrent Architecture

Several architectural variants have been proposed to make Transformer models recurrent, enabling adaptive test-time compute (Dehghani, et al. 2019, Hutchins, et al. 2022, Bulatov, et al. 2022). A comprehensive literature review would make this post excessively long, so we focus on only a few representative designs.

Universal Transformer (Dehghani, et al. 2019) combines Transformer self-attention with the recurrence mechanism of RNNs, dynamically adjusting the number of steps using adaptive computation time (Graves, 2016). At a high level, it can be interpreted as a recurrent function that learns token-wise hidden-state representations. When the number of steps is fixed, a Universal Transformer is equivalent to a multi-layer Transformer with parameters shared across layers.

A more recent recurrent design proposed by Geiping et al. (2025) adds a recurrent block $R$ on top of a standard Transformer. Each iteration of this recurrent block takes the embedding $\mathbf{e}$ and a random state $\mathbf{s}_i$. Conceptually, this recurrent-depth architecture resembles a conditioned diffusion model: the original input $\mathbf{e}$ is provided at every recurrent step, while a randomly initialized Gaussian state $\mathbf{s}_i$ is iteratively updated. (Notably, some experimental variants that more closely resembled diffusion models performed poorly.)

$ \begin{aligned} \mathbf{e} &= P(\mathbf{x}) & \text{embedding} \\ \mathbf{s}\_0 &\sim \mathcal{N}(\mathbf{0}, \sigma^2 I\_{n \cdot h}) \\ \mathbf{s}\_i &= R(\mathbf{e}, \mathbf{s}\_{i-1}) \quad\text{ for }i \in {1, \dots, r} & \small{\text{recurrent block; resembles a Transformer block}}\\ \mathbf{p} &= C(\mathbf{s}\_r) & \text{unembedding} \end{aligned} $

The recurrence count $r$ is randomized during training and sampled per input sequence from a log-normal Poisson distribution. To control compute cost, backpropagation is truncated to only the last $k$ iterations of the recurrent unit ($k=8$ in experiments), which enables training on the heavy-tail portion of the Poisson distribution. The embedding block continues to receive gradient updates at every step because its output $\mathbf{e}$ is injected each time, which mirrors RNN training. Unsurprisingly, training stability for recurrent models is highly sensitive. Initialization, normalization, and hyperparameters all matter, particularly at larger scales. For example, hidden states may collapse by predicting the same hidden state for every token, or the model may learn to ignore the incoming state $\mathbf{s}$. To improve stability, Geiping et al. used an embedding scale factor, a small learning rate, and careful tuning.

Thinking Tokens

Thinking tokens are implicit tokens introduced during training or inference that do not carry direct linguistic meaning. Instead, they provide additional thinking time and compute, helping the model achieve better performance.

Herel & Mikolov (2023) proposed inserting special thinking tokens (<T>) after every word in a sentence and training the model on this modified dataset. Each thinking token provides extra time for the model to process the context and make improved predictions. In a toy-model setup, training with thinking tokens yields lower perplexity than a baseline trained without them. The advantage of thinking tokens is stronger for non-trivial reasoning tasks or for sentences containing numbers.

Similarly, pause tokens proposed by Goyal et al. (2024) delay model outputs by appending dummy tokens (for example, characters such as . or #) to the end of the input sequence, which grants the model additional compute during inference. It is important to include pause tokens during both training and inference; fine-tuning with pause tokens alone produces limited gains. During training, multiple copies of pause tokens are inserted at uniformly random positions, and the loss on pause tokens is ignored.

Notably, thinking tokens and pause tokens in these experiments add neither additional information nor many new parameters. Why, then, do they help? First, they expand computation by introducing additional inference loops, effectively increasing computational capacity. Second, they can be interpreted as a special implicit form of CoT. One limitation is that the model must be pretrained to accommodate thinking tokens. Even so, this approach is an intriguing way to further improve utilization of test-time compute beyond inference-time CoT prompting.

Quiet-STaR (Zelikman et al. 2025) introduces token-level reasoning by training the model to generate rationales after every token to explain future text. It combines future-token predictions made with rationales and without rationales, learns how to produce better rationales, and uses REINFORCE to optimize rationale quality.

Quiet-STaR includes three stages:

• Think: Predict the next tokens with rationales. Because token-level reasoning is computationally expensive, the design generates multiple rationales in parallel. A specialized attention map ensures that each thought token attends only to itself, earlier thought tokens within the same thought, and the preceding text.

• Talk: Mix next-token prediction without rationales with the post-rationale prediction. The mixing weight between the two logits is learned via a special mixing head, implemented as a shallow MLP applied to the hidden output after each rationale. The correct next token can be selected with teacher forcing.

• Learn: Train the model to generate better rationales using REINFORCE, learning from examples that increase the probability of the correct next token and discarding those that degrade the prediction.

Without dataset-specific fine-tuning, Quiet-STaR improves zero-shot performance on CommonsenseQA (36.3%→47.2%) and GSM8K (5.9%→10.9%) in experiments using Mistral 7B.

Thinking as Latent Variables

A latent-variable model defines a probabilistic framework in which observed data is explained through unobserved (latent) variables. These latent variables represent hidden structure or intermediate processes that produce observable outcomes. Language models can be interpreted as probabilistic latent-variable models in which test-time thinking and reasoning steps correspond to latent thought variables (Zhou et al. 2020, Phan et al. 2023). Such a model defines a joint distribution over problems x_i, answers y_i, and latent thoughts z_i. We aim to optimize the log-likelihood of answers given questions, treating a range of CoTs as latent variables (N is the number of samples; $K$ is the number of CoTs per problem):

$ \begin{aligned} \log \mathcal{L}(\theta) &= \log p(y \mid x) \\ &= \log \sum_{k=1}^K p(y, z^{(k)} \mid x) \\ &= \log \sum_{k=1}^K p(z^{(k)} \mid x)\; p(y \mid z^{(k)}, x) \\ &= \log \mathbb{E}_{z^{(k)} \sim p(z^{(k)} \mid x)} \; p(y \mid z^{(k)}, x) \\ \end{aligned} $

The objective is to maximize the marginal likelihood of the correct answer, $p(y \mid x)$, given multiple reasoning traces per problem, $\{z^{(k)}\}_{k=1}^K$.

Expectation-Maximization

Expectation-Maximization is a widely used iterative method for optimizing model parameters when latent (hidden) variables are present. Accordingly, it can be used to train improved CoTs and then condition on them to generate stronger responses. In typical EM, we alternate between an E-step (Expectation), where we infer missing information about the latent variables (for example, how to sample improved CoTs), and an M-step (Maximization), where we optimize model parameters given the latent variables (for example, how to produce better answers), repeating until convergence.

$ \log \mathcal{L}(\theta) = \log \mathbb{E}_{\underbrace{z^{(k)} \sim p(z^{(k)} \mid x)}_\text{E-step}}\;\underbrace{p(y \mid z^{(k)}, x)}_\text{M-step} $

Because we cannot directly sample from the latent-variable distribution $p(z \mid x, y)$, researchers have explored approaches based on human-annotated data (Zhou et al. 2020), Metropolis-Hastings MCMC (Phan et al. 2023), and Monte Carlo sampling with specialized importance weights (Ruan et al. 2025) to obtain useful CoT samples for model updates. Ruan et al. (2025) investigated EM training on a large corpus of Web text augmented with latent thoughts, where latent thoughts are synthesized per chunk of observed data and the model learns autoregressively over both the latent thoughts and the observed data.

They first prompt a LLM $\tilde{q}$ to generate synthetic latent thought Z_i given observed data X_i:

You are provided with a pair of web document prefix and suffix. Your task is to insert latent thoughts between them underlying the creation of the suffix conditioned on the prefix. The latent thoughts should include: the missing background knowledge and the reasoning traces underlying each claim (especially, step-by-step derivations or logical reasoning).

Special tokens such as <StartOfLatent><Prior> ... <EndOfPrior> are used to insert the generated latent-thought content into raw data, enabling training of either the joint distribution $p(z, x)$ or the approximate posterior $q(z \mid x)$, depending on whether $z$ is inserted before or after $x$. However, because a LLM $\tilde{q}(z \mid x)$ is used to generate CoTs, this imposes a ceiling on the quality achievable by the approximate $q(z \mid x)$. Ruan et al. introduced importance weights to select CoT samples in the E-step, defined as:

$ w^{(k)} = \frac{p(z^{(k)}, x)}{q(z^{(k)} \mid x)} = \frac{p(x \mid z^{(k)}) \; p(z^{(k)})}{q(z^{(k)} \mid x)} $

, so that we prioritize samples with CoTs that predict the observation well (that is, high $p(x \mid z^{(k)})$), are simple and intuitive (that is, high $p(z^{(k)})$), and are informative without being overly obvious (that is, low $q(z^{(k)} \mid x)$).

Iterative Learning

Because pretrained models already can generate chains of thought, it is natural to build an iterative improvement loop that generates multiple CoTs and fine-tunes the model only on rationales that yield correct answers.
However, this naive approach can fail because the model receives no learning signal on problems it cannot solve. STaR (“Self-taught reasoner”; Zelikman et al. 2022) addresses this issue by adding a rationalization step for failed attempts. In this step, the model generates improved CoTs backward, conditioned on the problem and the ground-truth answer, enabling it to produce more plausible CoTs. The model is then fine-tuned on correct solutions, either those that originally led to correct outputs or those produced through rationalization.

STaR can be interpreted as an approximation to policy gradient in RL, using an indicator function as the reward, $\mathbb{1}[\hat{y} = y]$. We aim to maximize the expected reward when sampling $z \sim p(z \mid x)$ and then $y \sim p(y \mid x, z)$, since $p(y \mid x) = \sum_z p(z \mid x) \; p(y \mid x, z)$.

$ \begin{aligned} \nabla_\theta J(\theta) &= \nabla_\theta \mathbb{E}_{z_i, y_i \sim p(.\mid x_i)} \mathbb{1}[y_i = y_i^\text{truth}] \\ &= \sum_{i=1}^N \nabla_\theta \mathbb{1}[y_i = y_i^\text{truth}] \; p(y_i, z_i \mid x_i) \\ &= \sum_{i=1}^N \mathbb{1}[y_i = y_i^\text{truth}] \; p(y_i, z_i \mid x_i) \frac{\nabla_\theta p(y_i, z_i \mid x_i)}{p(y_i, z_i \mid x_i)} & \text{;log-derivative trick}\\ &= \mathbb{E}_{z_i, y_i \sim p(.\mid x_i)} \mathbb{1}[y_i = y_i^\text{truth}] \; \nabla_\theta \log p(y_i, z_i \mid x_i) & \text{;log-derivative trick} \end{aligned} $

Each iteration is equivalent to selecting CoT samples according to $\mathbb{1}[y=y^\text{truth}]$ and then applying supervised fine-tuning to increase the log probability of generating high-quality CoTs and answers. STaR performance improves with additional training iterations, and the rationalization step accelerates learning by generating better CoTs. The authors observed that high-temperature sampling increases the likelihood of producing correct answers paired with incorrect reasoning, and fine-tuning on such data can harm generalization. For datasets without ground-truth answers, majority voting over multiple high-temperature outputs can serve as a proxy for ground-truth labels (Wang et al. 2022), which enables training on synthetic samples.

Scaling Laws for Thinking Time

We have already seen substantial evidence that allowing models to use additional compute for reasoning before producing final answers at inference time can materially improve performance. Techniques such as prompting models to generate intermediate reasoning steps before answers, or training models to pause and reflect before predicting subsequent tokens, have been shown to improve performance beyond the capability limit achieved during training. This adds a new dimension for improving model intelligence, complementing established scaling-law factors such as model size, training compute, and data quantity (Kaplan et al. 2020).

Recent work suggests that optimizing LLM test-time compute may be more effective than scaling parameter count (Snell et al. 2024, Wu et al. 2025). Smaller models paired with sophisticated inference algorithms can achieve Pareto-optimal cost-performance trade-offs.

Snell et al. (2024) evaluated test-time compute relative to pretraining compute and found they are not exchangeable on a 1:1 basis. Test-time compute can readily close gaps on easy and medium problems when the underlying capability gap is small, but it is less effective on hard problems. The ratio of token budgets between pretraining and inference is crucial. Test-time compute is preferable only when inference tokens are substantially fewer than pretraining tokens. This highlights that building a strong base model with sufficient pretraining data and compute remains essential, because test-time compute cannot solve everything or compensate for large capability gaps.

s1 models (Muennighoff & Yang, et al. 2025) explored scaling CoT reasoning-path length using a budget forcing technique (that is, explicitly lengthening the path by appending "wait", or shortening it by terminating the thinking process using an end-of-thinking token or "Final Answer:"). They reported a clear positive correlation between average thinking time (measured in tokens) and downstream evaluation accuracy.

When comparing budget forcing to other decoding strategies that control reasoning-trace length, it is notable that simple rejection sampling (that is, sampling until the generation length fits within a token budget) produces reversed scaling, meaning longer CoTs lead to worse performance.

What’s for Future

Research on test-time compute and chain-of-thought reasoning creates new opportunities to strengthen model capabilities. More importantly, test-time thinking moves us toward AI systems that reflect best practices in human cognition, including adaptability, flexibility, critical reflection, and error correction. Progress so far should motivate further work to improve and deeply understand not only how, but also why, we (and our models) think.

In closing, I would like to encourage further research on the following open questions in test-time compute and chain-of-thought reasoning.

• Can we encourage models to produce human-readable, faithful reasoning traces during RL training while avoiding reward hacking?
• How should reward hacking be defined? Can we detect reward hacking during RL training or inference without human intervention? How can we avoid “whack-a-mole” fixes for reward hacking in RL training?
• Self-correction can occur within chain-of-thought, or it can be explicitly encouraged during multi-turn RL. How can we train models to self-correct without hallucination or regression when ground truth is unavailable?
• How can we run RL training with CoT rollouts for tasks that are highly contextual, personalized, and difficult to grade, such as creative writing, coaching, or brainstorming?
• In real deployments, test-time thinking cannot grow indefinitely. How can we transfer performance gains back into the base model while reducing inference cost (for example, via distillation)?
• How can we make test-time spending more adaptive to the difficulty of the current problem?

Citation

Please cite this work as:

Weng, Lilian. "Why We Think". Lil'Log (May 2025). https://lilianweng.github.io/posts/2025-05-01-thinking/

Or use the BibTex citation:

@article{weng2025think,
  title = {Why We Think},
  author = {Weng, Lilian},
  journal = {lilianweng.github.io},
  year = {2025},
  month = {May},
  url = "https://lilianweng.github.io/posts/2025-05-01-thinking/"
}

References

[1] Alex Graves. “Adaptive Computation Time for Recurrent Neural Networks.”. arXiv preprint arXiv:1603.08983 (2016).

[2] Wang Ling, et al. “Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems.”. arXiv preprint arXiv:1705.04146 (2017).

[3] Karl Cobbe, et al. “Training Verifiers to Solve Math Word Problems.”. arXiv preprint arXiv:2110.14168 (2021).

[4] Jason Wei, et al. “Chain of Thought Prompting Elicits Reasoning in Large Language Models.”. NeurIPS 2022.

[5] Maxwell Nye, et al. “Show Your Work: Scratchpads for Intermediate Computation with Language Models.”. arXiv preprint arXiv:2112.00114 (2021).

[6] Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux (2013).

[7] Takeshi Kojima, et al. “Large Language Models are Zero-Shot Reasoners.”. NeurIPS 2022.

[8] Michihiro Yasunaga, et al. “Large Language Models as Analogical Reasoners”. arXiv preprint arXiv:2310.01714 (2023).

[9] Eric Zelikman, et al. “STaR: Bootstrapping Reasoning With Reasoning.”. NeurIPS 2022.

[10] Xuezhi Wang, et al. “Self-consistency Improves Chain of Thought Reasoning in Language Models.”. ACL 2023.

[11] Ryo Kamoi, et al. “When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs.”. TACL 2024.

[12] Jie Huang, et al. “Large Language Models Cannot Self-Correct Reasoning Yet.”. ICLR 2024.

[13] Noah Shinn, et al. “Reflexion: Language Agents with Verbal Reinforcement Learning.”. arXiv preprint arXiv:2303.11366 (2023).

[14] Yunxiang Zhang, et al. “Small Language Models Need Strong Verifiers to Self-Correct Reasoning.”. ACL Findings 2024.

[15] Hao Liu, et al. “Chain of Hindsight Aligns Language Models with Feedback.”. arXiv preprint arXiv:2302.02676 (2023).

[16] Sean Welleck, et al. “Generating Sequences by Learning to Self-Correct.”. arXiv preprint arXiv:2211.00053 (2023).

[17] Yuxiao Qu, et al. “Recursive Introspection: Teaching Language Model Agents How to Self-Improve.”. arXiv preprint arXiv:2407.18219 (2024).

[18] Aviral Kumar, et al. “Training Language Models to Self-Correct via Reinforcement Learning.”. arXiv preprint arXiv:2409.12917 (2024).

[19] Hunter Lightman, et al. “Let’s Verify Step by Step.”. arXiv preprint arXiv:2305.20050 (2023).

[20] Yuxi Xie, et al. “Self-Evaluation Guided Beam Search for Reasoning.”. NeurIPS 2023.

[21] Yangzhen Wu, et al. “Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models”. ICLR 2025.

[22] Dongwei Jiang, et al. “RATIONALYST: Pre-training Process-Supervision for Improving Reasoning”. arXiv preprint arXiv:2410.01044 (2024).

[23] Xuezhi Wang and Denny Zhou. “Chain-of-Thought Reasoning Without Prompting.”. arXiv preprint arXiv:2402.10200 (2024).

[24] DeepSeek-AI. “DeepSeek-V3 Technical Report.” arXiv preprint arXiv:2412.19437 (2024).

[25] DeepSeek-AI. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.”. arXiv preprint arXiv:2501.12948 (2025).

[26] Luyu Gao, Aman Madaan & Shuyan Zhou, et al. “PAL: Program-aided Language Models.”. ICML 2023.

[27] Shunyu Yao, et al. “ReAct: Synergizing Reasoning and Acting in Language Models.”. ICLR 2023.

[29] Bowen Baker, et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.”. arXiv preprint arXiv:2503.11926 (2025).

[30] Wojciech Zaremba, et al. “Trading Inference-Time Compute for Adversarial Robustness.”. arXiv preprint arXiv:2501.18841 (2025).

[31] Tamera Lanham, et al. “Measuring Faithfulness in Chain-of-Thought Reasoning”. arXiv preprint arXiv:2307.13702 (2023).

[32] Boshi Wang, et al. “Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters.”. ACL 2023.

[33] Miles Turpin, et al. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.”. NeuriPS 2023.

[34] James Chua & Owain Evans. “Are DeepSeek R1 And Other Reasoning Models More Faithful?”. arXiv preprint arXiv:2501.08156 (2025).

[35] Yanda Chen et al. “Reasoning Models Don’t Always Say What They Think”. arXiv preprint arXiv:2505.05410 (2025).

[36] Edward Yeo, et al. “Demystifying Long Chain-of-Thought Reasoning in LLMs.”. arXiv preprint arXiv:2502.03373 (2025).

[37] Mostafa Dehghani, et al. “Universal Transformers.”. ICLR 2019.

[38] DeLesley Hutchins, et al. “Block-Recurrent Transformers.”. NeurIPS 2022.

[39] Aydar Bulatov, et al. “Recurrent Memory Transformers.”. NeuriPS 2022.

[40] Jonas Geiping, et al. “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.”. arXiv preprint arXiv:2502.05171 (2025).

[41] Herel & Mikolov. “Thinking Tokens for Language Modeling.”. AITP 2023.

[42] Sachin Goyal et al. “Think before you speak: Training Language Models With Pause Tokens.”. ICLR 2024.

[43] Eric Zelikman, et al. “Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking.”. arXiv preprint arXiv:2403.09629 (2025).

[44] Wangchunshu Zhou et al. “Towards Interpretable Natural Language Understanding with Explanations as Latent Variables.”. NeurIPS 2020.

[45] Du Phan et al. “Training Chain-of-Thought via Latent-Variable Inference.”. NeurIPS 2023.

[46] Yangjun Ruan et al. “Reasoning to Learn from Latent Thoughts.”. arXiv preprint arXiv:2503.18866 (2025).

[47] Xuezhi Wang et al. “Rationale-Augmented Ensembles in Language Models.”. arXiv preprint arXiv:2207.00747 (2022).

[48] Jared Kaplan, et al. “Scaling Laws for Neural Language Models.”. arXiv preprint arXiv:2001.08361 (2020).

[49] Niklas Muennighoff & Zitong Yang, et al. “s1: Simple test-time scaling.”. arXiv preprint arXiv:2501.19393 (2025).

[50] Peiyi Wang, et al. “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations” arXiv preprint arXiv:2312.08935 (2023).

[51] Yixin Liu, et al. “Improving Large Language Model Fine-tuning for Solving Math Problems.” arXiv preprint arXiv:2310.10047 (2023).

[52] Charlie Snell, et al. “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.”. arXiv preprint arXiv:2408.03314 (2024).

[53] OpenAI. o1-preview: “Learning to reason with LLMs.” Sep 12, 2024.

[54] OpenAI. o3: “Introducing OpenAI o3 and o4-mini.” Apr 16, 2025.