The economist Thomas Sowell famously remarked that "there are no solutions, only trade-offs." In no field is this axiom more painfully (and expensively) visible today than in the engineering of frontier artificial intelligence. Before a laboratory turns on its GPU clusters for a multi-million-dollar training run, it must answer what is arguably the most expensive question in computer science: given a fixed budget of compute, should the capital buy a larger model, or should it buy more data to train a smaller one?
An incorrect answer costs tens of millions of dollars and consigns a model to irrelevance. The correct answer has changed twice in the last six years, and understanding why is the closest thing the field has to a general theory of its own economics.
The Kaplan Recipe: Parameters Over Data
The first attempt at a general theory of scaling came in January 2020, when Jared Kaplan and his colleagues at OpenAI published “Scaling Laws for Neural Language Models.” Sweeping across seven orders of magnitude of compute, they found that a model’s cross-entropy loss followed clean power laws. The architectural details (width, depth, number of heads) were largely irrelevant within a broad range. The critical factors were parameters \(N\), dataset size \(D\), and total compute \(C\).
Kaplan’s most consequential conclusion was operational. For a given compute budget, the optimal allocation of resources favored model size over data volume:
\[N_{\text{optimal}} \propto C^{0.73}, \qquad D_{\text{optimal}} \propto C^{0.27}\]This relationship translated into a simple industry slogan: larger models are radically more sample-efficient. The operational advice was to buy parameters and stop training well short of convergence. The industry followed this advice. GPT-3 (175B parameters) was trained on roughly 300 billion tokens—a ratio of only 1.7 tokens per parameter. Gopher (280B) and Megatron-Turing NLG (530B) followed the same recipe.
The Chinchilla Correction: Data Was Under-Ordered
In March 2022, Jordan Hoffmann and the DeepMind team challenged this orthodoxy. In their paper, “Training Compute-Optimal Large Language Models,” they held compute budgets constant and swept the ratio of \(N\) to \(D\) using the "IsoFLOP" methodology. Their U-shaped performance curves showed that model size and dataset size should scale in equal proportion:
\[N_{\text{optimal}} \propto C^{0.50}, \qquad D_{\text{optimal}} \propto C^{0.50}\]This yielded the now-canonical rule of thumb: a compute-optimal model requires roughly 20 tokens per parameter. To prove their thesis, DeepMind trained Chinchilla, a 70B parameter model, on 1.4 trillion tokens using the same compute budget as the 280B Gopher. Chinchilla outperformed Gopher, GPT-3, and other larger models across the board. The implication was clear: the industry had spent billions of dollars training models that were, in Hoffmann’s polite formulation, "significantly undertrained."
Funnily enough, the technical discrepancy between Kaplan and Chinchilla was not resolved by algorithmic breakthroughs, but by a simple correction in counting. In June 2024, Tim Pearce and Jinyeop Song demonstrated that Kaplan’s steeper scaling exponent was a counting artifact. Kaplan had measured $N$ using only non-embedding parameters. At small training runs, embedding tables consume a large fraction of the total parameter count, distorting the slope. Once the embedding parameters are accounted for, both approaches converge on the Chinchilla optimal ratio.
Interactive Scaling Law Calculator
To visualize how these scaling regimes allocate compute, and how inference volume shifts the optimal trade-off, use the interactive calculator below. Adjust the training compute budget and the expected query volume to see how the optimal model size and token count adapt.
Lifecycle Compute Optimizer
| Scaling Regime | Model Size (N) | Dataset Size (D) | Tokens/Param Ratio |
|---|---|---|---|
| Kaplan (2020) | -- | -- | -- |
| Chinchilla (2022) | -- | -- | -- |
| Amortized Inference | -- | -- | -- |
The Amortized Inference Shift
Yet, if you look at the open-weights landscape today, almost nobody obeys the Chinchilla-optimal 20:1 ratio.
- Meta’s Llama 3 8B was trained on over 15 trillion tokens—a ratio of 1,875 tokens per parameter.
- Alibaba’s Qwen 2.5 7B saw 18 trillion tokens—a ratio of over 2,570 tokens per parameter.
- DeepSeek V3 pretrained on 14.8 trillion tokens, representing roughly 400 tokens per active parameter.
This deliberate "overtraining" does not contradict Chinchilla. Rather, it optimizes for a different objective function. Hoffmann’s equations minimized the compute required to train a model. They ignored the cost of running that model in production after training ends.
Once a model is deployed to millions of users, the lifetime cost of serving inference dwarfs the upfront cost of training. A smaller, overtrained model requires fewer GPUs to run, saving millions in serving infrastructure (which is why our own supply chain algorithms are constantly being pruned for execution speed). Overtraining is training-compute suboptimal, but it is highly compute-rational over the model’s operational lifespan.
A recent Wisconsin–Stanford paper, Test-Time Scaling Makes Overtraining Compute-Optimal, has formalized this by adding a third axis—inference-time samples—and finds that when reasoning models like o1 or DeepSeek-R1 will be sampled hundreds of times per query, the optimum shifts radically into the overtrained regime, beyond the range most pretraining studies even measure. In our analysis of recursive AutoResearch loops, we saw how stacking reasoning loops at inference time routes around model constraints, reinforcing the value of highly optimized base models.
The Densing Law and Capability Density
This structural efficiency is validated by what researchers call capability density—the ratio of the "effective parameter size" (how large an older model would need to be to match a modern model’s benchmarks) to the actual parameter size.
In late 2024, a team from Tsinghua and OpenBMB introduced the Densing Law, documenting that the capability density of open-weights models has doubled roughly every 3.5 months.
| Model | Release Date | Parameter Count | MMLU Score | Relative Capability Density |
|---|---|---|---|---|
| LLaMA 1 | Feb 2023 | 65B | 68.9% | 1.0x (Baseline) |
| Qwen 2.5 | Sep 2024 | 72B | 85.3% | 4.8x |
| Llama 3.1 | Jul 2024 | 70B | 86.0% | 5.1x |
This exponential increase explains why a modern 72B model can match or exceed a 405B model trained just a year prior. It is not luck; it is the compounding impact of superior data curation, better tokenizers, and structural architectural refinements like Grouped-Query Attention (GQA) and SwiGLU activations.
The Shrinking Fine-Tuning Headroom
This density shift has practical implications for enterprise adaptation. Base models today arrive already saturated with post-training. Qwen 2.5 underwent supervised fine-tuning on over a million instruction samples, followed by multi-stage reinforcement learning with both offline DPO and online GRPO.
Because the base model has already had its low-hanging fruit harvested, generic instruction fine-tuning now yields diminishing marginal returns. Downstream teams should focus their efforts away from general fluency and toward domain-specific adaptation (e.g., specialized legal reasoning, medical coding, or unique brand voices) where the general-purpose pretraining distribution remains sparse. Fortunately, parameter-efficient fine-tuning (PEFT) methods like LoRA have reduced these costs to the point where the business question is no longer whether to fine-tune, but what domain data to prioritize.
In our upcoming analysis of Lean Six Sigma in AI, we will explore how these overtrained models redefine the cost of quality in the corporate value stream.
Ultimately, scaling laws are not immutable laws of physics. They are contingent, empirical, and sensitive to what we count, how we measure it, and how we plan to deploy the system. The labs winning the market today are not those with the largest compute clusters, but those who read the trajectory of these trade-offs the fastest.
Victor Blancada is a data scientist focused on deriving actionable insights for clients. Visit his LinkedIn page here.