Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
For deploying foundation models, practitioners increasingly need prescriptive scaling laws: given a pre training compute budget, what downstream accuracy is attainable with contemporary post training practice, and how stable is that mapping as the field evolves? Using large scale observational evaluations with 5k observational and 2k newly sampled data on model performance, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate the temporal reliability by fitting on earlier model generations and evaluating on later releases. Across various tasks, the estimated boundaries are mostly stable, with the exception of math reasoning that exhibits a consistently advancing boundary over time. We then extend our approach to analyze task dependent saturation and to probe contamination related shifts on math reasoning tasks. Finally, we introduce an efficient algorithm that recovers near full data frontiers using roughly 20% of evaluation budget. Together, our work releases the Proteus 2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.
699550588d17d1ee8c10ecd7
Hanlin Zhang
admin_assigned
2026-02-18T13:45:20.828Z
624054bcc2c17da6a63eb539
/avatars/bf52dc0683b4100733f8696a97696d0e.svg
hlzhang109
hlzhang109
user
hlzhang109
699550588d17d1ee8c10ecd8
Jikai Jin
699550588d17d1ee8c10ecd9
Vasilis Syrgkanis
admin_assigned
2026-02-18T13:45:26.632Z
690526084ae4c0fc109fbaef
/avatars/87e1257024d79c4469dad48a8bb3bdab.svg
Vasilis Syrgkanis
vsyrgk
user
vsyrgk
699550588d17d1ee8c10ecda
Sham Kakade
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
For deploying foundation models, practitioners increasingly need prescriptive scaling laws: given a pre training compute budget, what downstream accuracy is attainable with contemporary post training practice, and how stable is that mapping as the field evolves? Using large scale observational evaluations with 5k observational and 2k newly sampled data on model performance, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate the temporal reliability by fitting on earlier model generations and evaluating on later releases. Across various tasks, the estimated boundaries are mostly stable, with the exception of math reasoning that exhibits a consistently advancing boundary over time. We then extend our approach to analyze task dependent saturation and to probe contamination related shifts on math reasoning tasks. Finally, we introduce an efficient algorithm that recovers near full data frontiers using roughly 20% of evaluation budget. Together, our work releases the Proteus 2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.
Large-scale observational analysis estimates capability boundaries and performance predictions for foundation models using quantile regression and evaluates temporal stability across tasks.
scaling laws quantile regression sigmoid parameterization conditional quantiles observational evaluations model performance compute budget capability boundaries temporal reliability smoothing