Scaling laws are among the most important empirical discoveries in deep learning. The core observation is straightforward: the training loss $L$ decreases in a highly regular way as we increase model size $N$, dataset size $D$, and compute $C$. The relationship follows a power-law curve, which appears as a straight line in a log-log plot. More generally, scaling laws provide a framework for characterizing how compute, loss, model size, and data interact. At their core, they address how to allocate scarce compute optimally between $N$ and $D$.

This predictability makes scaling laws practically valuable. A common workflow is to fit scaling laws using a small number of small-scale training runs, then extrapolate to estimate the token and compute requirements for much larger models.

Early days: ML loss predictability

The idea that generalization error can be predicted as a function of scale was studied well before “scaling laws” became a mainstream term.

Amari et al. (1992) derived four types of learning curves using a Bayesian method and the annealed approximation.

1. Deterministic learning algorithm, noiseless data, one unique solution: $\epsilon \sim c \cdot D^{-1}$, where $c$ is a constant.
2. Deterministic learning algorithm, noiseless data, multiple equivalent solutions: $\epsilon \sim c \cdot D^{-2}$. Learning accelerates with each additional data point because the model only needs to learn the optimal parameter manifold rather than locate a single solution point.
3. Deterministic learning algorithm, noisy data: $\epsilon \sim c \cdot D^{-1/2}$. Noise in the data makes learning more difficult.
4. Stochastic learning algorithm, noisy data: $\epsilon \sim c \cdot D^{-1} + E$. In this setting, the irreducible loss $E$ is the residual error that a stochastic learner cannot reduce further, for example, when the model exhausts its capacity on large datasets. All four learning-curve types follow a power law:

where $E$ can be 0 and $\alpha = -2, -1, -1/2$. Although the theoretical setup is based on a simplified binary classification task, it provides a useful direction for developing empirical ML loss prediction models.

One of the earliest empirical investigations by Hestness et al. (2017) analyzed the relationship between generalization error, model size, and data. For a fixed training dataset size, they identified the best-fit model size via grid search, then plotted loss versus training dataset size. Across four deep learning domains (neural machine translation, image classification, language modeling, and speech recognition), they observed a recurring pattern:

• Generalization error scales as a power law across a set of factors (for example, data size).
• Model improvements shift the error curve but do not appear to change the power-law exponent.
• Notably, architecture affects the offset ($E$) of the power-law fit but does not change the exponent ($\alpha$). The power-law slope appears to be a property of the problem domain rather than the model architecture.
• The number of model parameters $N$ required to fit a dataset of size $D$ also scales as a power law.

A conceptual diagram separates the learning curve into three phases. In the small-data phase, when learning signals are insufficient, performance is only slightly better than random guessing. In the middle phase (the “power-law region”), loss exhibits a power-law relationship with data and model size. The final irreducible-error phase can be attributed to factors such as noise in the data.

Rosenfeld et al. (2020) extended this line of work by modeling error as a joint function of model size $N$ and data size $D$ across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants). Empirically, they found that if one axis is held fixed, error decays as a power law in the other:

These relationships can be combined into a joint form:

where $A > 0, B > 0, \alpha \geq 0, \beta \geq 0$ are scalar constants and $E$ does not depend on either $N$ or $D$.

As a result, they can construct a predictive model as a simple parametric function with $\boldsymbol{\theta} = \langle A, B, E, \alpha, \beta \rangle$, enabling prediction of the expected loss for $(D, N)$ > certain thresholds using training data drawn only from smaller configurations, $(D, N)$ < certain thresholds.

Side note: These early works draw on classical learning-theory intuition such as the VC dimension (the cardinality of the largest set of points a model can shatter) as a proxy for capacity. In modern deep learning, however, the VC dimension is often too coarse to explain observed behavior, and empirical power laws have proven cleaner and more operational than the worst-case theoretical bounds.

Scaling Laws in Data-Infinite Region

Kaplan et al.’s Scaling Laws

Kaplan et al. (2020) popularized scaling laws within the language modeling community. They reported that the cross-entropy test loss $L$ scales as a power law with each of model size $N$ (excluding embedding layers), dataset size $D$, and training compute $C$, spanning many orders of magnitude. These results align with earlier work discussed above, but Kaplan et al. formalized the concept with a specific emphasis on Transformer language models and larger-scale empirical studies: model sizes ranged from 768M to 1.5B non-embedding parameters, and datasets ranged from 22M to 23B tokens. All training runs used a learning-rate schedule consisting of a 3000-step linear warmup followed by cosine decay to zero.

Key findings include:

• The loss $L$ scales as a power law with $N$, $D$, and $C$ individually. For optimal performance, all three must scale together.
• Training curves follow predictable power laws whose parameters are approximately independent of model size.
• Larger models are more sample-efficient, meaning they achieve a given loss with fewer optimization steps and fewer data points than smaller models.
• Architectural details (width, aspect ratio, and similar choices) matter less than overall scale.
• Train loss and test loss are positively correlated. (This may sound obvious, but it underpins pretraining practice. By contrast, whether improvements in pretraining loss transfer to downstream evaluation requires separate investigation.)
• With a fixed compute budget, it is more efficient to train a very large model and stop before convergence than to train a smaller model to convergence. This is the point at which the Chinchilla scaling laws (next section) disagree: Kaplan et al. overestimated the compute-optimal model size because their fitted exponent was larger.

They summarize the joint dependence on $N$ and $D$ with a single equation:

A useful implication of this form is that the degree of overfitting (that is, a complex model or a small dataset) depends primarily on the ratio $N^{\alpha / \beta} / D$. This suggests that dataset size must increase in a particular proportion to model size in order to avoid data-limited training.

The most influential (and, in retrospect, most disputed) conclusion concerned compute-optimal allocation. Kaplan et al. found $N_\text{opt} \propto C^{0.73}$ and concluded that model size should increase faster than dataset size. Concretely, for a 10x increase in compute, they recommended scaling model size by approximately 5.5x while increasing training tokens by only about 1.8x. The Chinchilla paper later overturned this recommendation, arguing that it leaves large models substantially undertrained.

Kaplan et al. also provided a practical approximation for the number of training FLOPs based on $D$ and $N$. Each multiply-add is counted as approximately 2 FLOPs.

Under a standard configuration where $d_\text{attn} = d_\text{model} = d_\text{ff}/4$, and excluding embedding layers from $N$ and the per-token forward compute:

They then treat backward-pass FLOPs as twice the forward-pass FLOPs, because backpropagation performs two matrix multiplications, one for gradients with respect to the input activations and one for gradients with respect to the weights. Therefore, total training FLOPs per token are approximately $6N$, and the total training FLOPs over $D$ tokens are $C \approx 6ND$.

Chinchilla Scaling Laws

The Chinchilla paper (Hoffmann et al. 2022) analyzed the relationship between compute-optimal model size $N$ (total parameters, including embeddings) and the number of training tokens $D$ under a fixed compute budget $C$. With a more careful experimental design, it reached conclusions that differ somewhat from Kaplan et al..

The central question is how to allocate resources under the constraint $\text{FLOPs}(N, D) = C \approx 6ND$. Put differently, given limited FLOPs (a fixed number of GPUs running for a fixed time), how should we trade off between more training tokens and more model parameters?

The Chinchilla paper introduced three carefully structured approaches for fitting scaling laws.

The experiments covered more than 400 models, spanning 70M to over 16B parameters and 5B to 500B training tokens. These experiments assume that every training token is unique (the infinite-data regime). All runs used a cosine learning-rate schedule that decays by 10x over the full training horizon. Sweeping over model sizes traces the compute-optimal frontier.

Method 1: Fix model sizes, vary the token budget

For each parameter count $N$, train multiple runs with different token budgets, then record the minimal loss achieved per FLOP budget $C$.

Method 2: IsoFLOP profiles

Fix a compute budget $C$ and plot final loss versus parameter count $N$. Each iso-FLOP curve is approximately parabolic in log space, and the minimum identifies the optimal model size for that compute budget. Repeating this across compute budgets traces out a power-law line in the resulting plot.

Method 3: Parametric fit

Fit the same parametric function as in Rosenfeld et al. (2020) directly,

A closed-form approximation for the optimal $N_\text{opt}(C), D_\text{opt}(C)$ can be obtained by minimizing $\hat{L}(N, D)$ subject to the constraint $\text{FLOPs}(N,D) = C \approx 6ND$.

First, reduce the expression so that it contains only $N$:

When $\alpha \approx \beta$, model size and training tokens should scale at the same rate.

To estimate the optimal $\boldsymbol{\theta} = \langle A, B, E, \alpha, \beta\rangle$, the Chinchilla paper uses a Huber loss (robust to outliers; $\delta=10^{-3}$) and the L-BFGS algorithm (effective for curve fitting with a small number of parameters).

Chinchilla derives its conclusion using three complementary methods whose final results agree, and this convergence is part of what made the result persuasive.

The Chinchilla paper’s claim that many large models (at the time, around ~2022) were undertrained is supported by a well-known demonstration: with the same compute budget as Gopher (Rae et al. 2021; 280B parameter count, 300B token budget), they trained Chinchilla (70B parameter count, 1.4T token budget). This model is 4x smaller but trained on roughly 4x more tokens, and it outperformed Gopher across the board.

Reconciling Kaplan and Chinchilla

The Chinchilla scaling laws differ from Kaplan et al. in the following ways:

• Rather than “grow the model faster than the data” ($N_\text{opt} \propto C^{0.73}$), for every doubling of model size you should also double the number of training tokens ($N_\text{opt} \propto C^{0.5}$).
• Rather than “train a big model and stop before convergence,” the recommendation becomes to train a smaller model on more data.

Both papers share the same underlying principle, but they disagree about where the compute-optimal tradeoff between model size and training tokens lies. Why is the disagreement so large?

Difference 1: Kaplan et al. primarily studied smaller models. Kaplan et al. ran experiments mostly in smaller-model regimes, whereas the Chinchilla experiments extended to more than 10x larger scales. In log-log extrapolation, even a small change in a fitted relationship can produce large downstream differences (see toy simulation).

Difference 2: Embedding parameter count is significant for small models. In the low-parameter regime, embedding parameters constitute a non-trivial fraction of total parameters, so whether they are included materially affects the analysis. Pearce & Song (2024) provides an in-depth treatment of this issue. Let $N_{\setminus E}, C_{\setminus E}$ denote model size and compute when embeddings are excluded, and let $N, C$ denote total parameters.

• Kaplan et al.: $N^*_{\setminus E} \propto C^{0.73}_{\setminus E}$ (non-embedding)
• Chinchilla: $N^* \propto C^{0.50}$ (total)

To connect these conventions, they fit a relationship between total parameters $N_T$ and non-embedding parameters $N_{\setminus E}$, for a constant $\omega$:

This form is strictly increasing and $\lim_{N \to \infty} N = N_{\setminus E}$ (because $\frac{N}{N_{\setminus E}} = 1 + \omega {N_{\setminus E}}^{- \frac{2}{3}}, \lim_{N_{\setminus E} \to \infty} \frac{N}{N_{\setminus E}} = 1$.

Substituting this into the Chinchilla scaling-law equation,

the relationship between $C_{\setminus E}$ and $N_{\setminus E}$ is no longer a clean power law. It can only be approximated locally as $N^*_{\setminus E} \overset{\propto}{\sim} C_{\setminus E}^g$, where $g$ is a local exponent derived from a first-order derivative ($\overset{\propto}{\sim}$) rather than a global power-law exponent, yielding $g = \frac{\mathrm{d} \log C_{\setminus E}}{\mathrm{d} \log N_{\setminus E}}$. For full details on how the exponent $g$ is approximated, see Appendix A.1 in Pearce & Song (2024).

As the visualization indicates, as $C_{\setminus E}$ increases, $g$ converges to the Chinchilla estimate. By generating synthetic training curves from the equation above, in the model-size range 768M to 1.5B (as in Kaplan et al.), they estimated that $g$ is close to Kaplan’s coefficient of 0.73 in that region.

Why power law?

Power laws appear across many domains beyond AI, including Zipf’s law, scale-free networks, urban scaling laws, and many other complex systems. A typical pattern is that large events are rare, small events are common, and the size-frequency relationship often becomes a straight line on a log-log plot.

Why do LLM scaling laws also take a power-law form?

Motivated in part by the fact that different domains exhibit different exponents (Hestness et al. 2017), an early explanation by Sharma & Kaplan (2020) proposes viewing language modeling as regression on a low-dimensional data manifold. Increasing model parameters enables a finer partition of this manifold and thus reduces generalization error. In the simplest framing, if a model of effective size $N$ partitions a $d$-dimensional manifold into $O(N)$ regions, the typical linear resolution scales like $\sim N^{-1/d}$. This yields a power-law form similar to the scaling laws above. The theory applies most cleanly in the infinite-data, underfitting regime, but in practice the intrinsic dimensionality of a data manifold is difficult to estimate.

A later hypothesis (Michaud et al. 2023, Brill 2024) assumes that knowledge or skills are acquired in discrete (“quantized”) units, and that the frequency distribution of these skills follows a power law. Under this view, the model learns common skills earlier and rarer skills later, producing a smooth power-law decay in loss.

Only two hypotheses are listed here, but other studies aim to explain power-law scaling via spectral tails in data, kernel eigenvalues, natural-language statistics, or phase transitions in training dynamics.

Scaling Laws in Data-Limited Region

Classic scaling laws effectively assume unlimited unique data, no repetition, and no multi-epoch training. As model sizes increase substantially, we begin to exhaust high-quality unique tokens. Some arguments about how long AI scaling can continue center on whether we are approaching a “data wall.”

It is also important to underscore that the dataset underlying $D$ is presumed to have been cleaned in advance. In practice, the pretraining data pipeline is often a major contributor to an effective pretraining setup. Common steps include deduplication (both exact and fuzzy), quality filtering, boilerplate removal, safety filtering, PII and copyright masking, benchmark decontamination, and careful reweighting of data-mixture components by language, quality, content type, and related factors. Even when two datasets have the same token count $D$, a high-quality corpus and a corpus of low-quality Internet content can produce dramatically different compute efficiency.

The study by Hernandez et al. (2022) examined a controlled setting: a mostly unique dataset containing only a small fraction of repeated data. Starting from a large dataset, their data mixture retains 90% non-repeated content while replacing the remaining 10% with repeats of a very small subset of the original. Training a Transformer for 100B tokens, they observed a double-descent phenomenon. In other words, test loss can become worse and then improve again as a function of how strongly the repeated data is emphasized. This effect becomes more pronounced as the repeated fraction increases.

The flat, or even rising, segment in the middle of training may be attributable to memorization of repeated data. Learning curves with this shape reduce the accuracy of scaling-law fits. The authors also concluded that repeated data degrades some OOD evaluation and downstream fine-tuning. That said, their mixture is assembled in a more laboratory-style setup, whereas repetition in real-world corpora is often subtler (for example, different sources exhibit different degrees of repetition, semantic repetition, and so on).

Rather than asserting that repetition necessarily harms training, the more relevant question here is how to fit scaling laws given that high-quality unique data is finite and some degree of repetition during training is likely unavoidable.

Muennighoff et al. (2023) addressed the question of how to allocate compute optimally when model training is constrained by available data. Specifically, they empirically evaluated the effects of data repetition across roughly 400 experiments, spanning 10M–9B parameters, data sizes up to 900B tokens, and up to 1500 epochs. In their setup, the exact same dataset is replayed each epoch, shuffled between epochs, and evaluated against a held-out test set.

The central modeling change is to decompose the total token count $D$ into two components: (i) the number of unique tokens $U_D$ and (ii) the number of repeats $R_D$ (that is, num. epochs - 1). This yields $D = U_D(1 + R_D)$. With a unique-data budget $D_\text{uniq}$, by definition $U_D = \min \{{ D_\text{uniq}, D\}}$ and $R_D = (D / U_D) - 1$. They apply the Chinchilla scaling laws to identify the optimal model size $U_N$ for fitting $U_D$, and they define excess model size induced by repeats as $R_N = (N / U_N) - 1$.

They then modify the Chinchilla parametric fit (method 3) by substituting effective (discounted) data $D’$ and effective model size $N’$ in place of the raw quantities:

The intuition is that the value of a token decays exponentially as it is repeated. In their formulation, each repetition reduces the token’s remaining value by a $(1 - 1/r_D)$ fraction, where $r_D$ is a learnable “half-life” parameter. When $R_D = 0$ or $R_D \ll r_D$, we recover $D’ \approx D$.

A symmetric construction is also used for excess model size, $N’ = U_N + U_N r_N(1 - \exp(-R_N / r_N))$, reflecting the idea that “larger models overfit more quickly on repeated data” and that “a model can be too large for its dataset.” This component is less straightforward, and I could not find a satisfactory explanation for why model size must enter in a symmetric manner analogous to repeated data. Subsequent work by Lovelace et al. (2026) revised this assumption.

Their empirical fit indicates that excess parameters decay faster in value than repeated data, $r_N < r_D$, implying that compute should be allocated more toward additional epochs rather than additional parameters. A limitation of this modeling approach, as the authors also noted, is that it substantially underestimates the final test loss for failing models (that is, models whose loss rises mid-training), such as models trained for 44 epochs.

More recently, Lovelace et al. (2026) returned to the same problem using a different approach. Instead of treating overparameterization as diminishing returns on effective model size, Lovelace et al. model the interaction between model size $\times$ and data repetition explicitly. Empirically, they trained about 300 models, spanning 15M to 1B parameters and 50M to 6B unique tokens.

When they plot the fit residual, holding model size fixed while varying the degree of data repetition, the pattern is intuitive: additional epochs do more harm. Notably, larger models are more sensitive to repetition. This suggests that the loss penalty likely depends on both model size and data size.

They introduce an explicit overfitting-penalty term structured around the capacity ratio $N / U_D$ (parameters relative to unique tokens):

where:

• $R_D$ is the repetition count;
• the scalar $P$ is a learnable parameter;
• the exponent $\kappa$ (the 2nd learnable parameter) allows the penalty to scale nonlinearly with the capacity ratio $N / U_D$;
• the separate exponent $\delta$ (the 3rd learnable parameter) on repetition decouples repetition nonlinearity from $\kappa$.

The additional term (shown in red) is a direct overfitting penalty that increases with both the number of times the data is repeated and the degree to which the model is overparameterized relative to the available unique data.

They also present a case study on the effect of weight decay under limited-data constraints, finding that stronger weight decay reduces the overfitting penalty associated with data repetition.

Because both the Muennighoff et al. and Lovelace et al. approaches are derived from empirical curve fitting, it remains unclear why data-constrained scaling laws should take precisely these functional forms and why each free parameter is necessary. I am interested in seeing more theoretical work in this direction.

Trickiness of Fitting Scaling Laws in Reality

Even though the functional form is clean, scaling-law fitting can be unexpectedly sensitive in practice to seemingly minor procedural choices, such as how parameters are counted, how numerical precision is rounded, and whether loss is summed or averaged.

A scaling law is typically fit using the (relatively small, relatively inexpensive) models we can afford to train, and then the result is extrapolated to models that are orders of magnitude larger. In this regime, choices that appear to be mere rounding details can lead to large differences in the final prediction.

At the same time, scaling-law fitting assumes that the only changing factor is scale. This implies that the architecture, optimizer, learning-rate schedule, batch ramp, data mixture, tokenizer, and other design choices should remain fixed. A further implicit assumption is that these settings have been appropriately tuned, since issues such as undertrained models can lead to different conclusions.

The disagreement between the results of Kaplan et al. and Chinchilla is one example that illustrates how tricky scaling-law fitting can be.

A second example is a follow-up analysis investigating why Chinchilla method 3 deviates slightly from the other two methods. Besiroglu et al. (2024) extracted the raw $(N, D, L)$ points from Figure 4 of Hoffmann et al. (2022) and re-ran the method 3 parametric fit. They identified several concrete issues:

• An excessively high loss scale in the L-BFGS-B minimizer, caused by averaging Huber-loss values across examples rather than summing them, which resulted in premature termination of the optimization. Early stopping of loss minimization in both the original fit and in bootstrapping produced inconsistent estimates and implausibly narrow confidence intervals.
• The reported $\alpha$ and $\beta$ were rounded to two digits of precision, making the derived $A, B$ appear more inaccurate than it actually was.

Toy simulation

Below is a toy simulation widget, created by ChatGPT, intended to demonstrate three specific failure modes.

We assume the ground-truth function is:

and therefore $N_\text{opt} \propto C^{0.5126}, D_\text{opt} \propto C^{0.4874}$. This is the estimate from Besiroglu et al. (2024).

The simulation plots the predicted loss $\hat{L}$ versus dataset size $D$, and provides a set of sliders that illustrate:

• Loss precision: rounding losses from high to low decimal precision can change the fitted parameter values.
• Loss noise: perturbing loss values by only a multiplier of milli-loss (0.001) units yields a different fit.
• Fit-region sensitivity: fitting only small models, only medium models, or all models produces different apparent scaling laws.

Citation

Please cite this work as follows:

Weng, Lilian. “Scaling Laws, Carefully”. Lil’Log (Jun 2026). https://lilianweng.github.io/posts/2026-06-24-scaling-laws/

Alternatively, use the BibTex citation below:

@article{weng2026scaling,
 title = {Scaling Laws, Carefully},
 author = {Weng, Lilian},
 journal = {lilianweng.github.io},
 year = {2026},
 month = {June},
 url = "https://lilianweng.github.io/posts/2026-06-24-scaling-laws/"
}

References

[1] S. Amari, N. Fujita, and S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.

[2] Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.

[3] Rosenfeld et al. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.

[4] Kaplan et al. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.

[5] Hoffmann et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.

[6] Pearce and Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.

[7] Bahri et al. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.

[8] Sharma and Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.

[9] Hernandez et al. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.

[10] Muennighoff et al. “Scaling Data-Constrained Language Models.” NeurIPS 2023.

[11] Lovelace et al. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.

[12] Besiroglu et al. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.

[13] Michaud et al. “The Quantization Model of Neural Scaling” NeurIPS 2023.

[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.

[15] Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.