Test-Time Scaling Makes Overtraining Compute-Optimal
Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test (T^2) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. T^2 modernizes pretraining scaling laws with pass@k modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from T^2 are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that T^2 scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making T^2 scaling meaningful in modern deployments.
69d3d7e02c523d36431faed2
Nicholas Roberts
69d3d7e02c523d36431faed3
Sungjun Cho
69d3d7e02c523d36431faed4
Zhiqi Gao
69d3d7e02c523d36431faed5
Tzu-Heng Huang
69d3d7e02c523d36431faed6
Albert Wu
69d3d7e02c523d36431faed7
Gabriel Orlanski
69d3d7e02c523d36431faed8
Avi Trost
69d3d7e02c523d36431faed9
Kelly Buchanan
claimed_verified
2026-04-07T08:37:28.428Z
62a8b52f4adaa29ed070181b
/avatars/fe61cab215248132f5ef01c5980c642e.svg
Kelly Buchanan
ekellbuch
user
ekellbuch
69d3d7e02c523d36431faeda
Aws Albarghouthi
69d3d7e02c523d36431faedb
Frederic Sala
Test-Time Scaling Makes Overtraining Compute-Optimal
Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test (T^2) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. T^2 modernizes pretraining scaling laws with pass@k modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from T^2 are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that T^2 scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making T^2 scaling meaningful in modern deployments.
Train-to-Test scaling laws jointly optimize model size, training tokens, and inference samples under fixed budgets, revealing that optimal pretraining decisions shift into overtraining regimes when inference costs are considered.
LLMs pretraining scaling laws test-time scaling pass@$k$ modeling joint optimization overtraining downstream tasks post-training