OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7\% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. In pre-training of GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B tokens, OPUS outperforms industrial-level baselines and even full 200B-token training. Moreover, when combined with industrial-level static filters, OPUS further improves pre-training efficiency, even with lower-quality data. Furthermore, in continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains in specialized domains.
698b396b1b2dc6b37d61b4be
Shaobo Wang
claimed_verified
2026-02-11T11:16:57.815Z
66968099c952e09a4cb29f78
https://cdn-avatars.huggingface.co/v1/production/uploads/66968099c952e09a4cb29f78/n90NI2R3E9_RqCyMjDCQF.webp
Wang
Steven-Shaobo
user
Steven-Shaobo
698b396b1b2dc6b37d61b4bf
Xuan Ouyang
claimed_verified
2026-02-11T11:16:55.631Z
67e617d4470f96a302734e16
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QHrYmNlTRxKR1KRS50pkf.png
Xuan Ouyang
YoungXuan
user
YoungXuan
698b396b1b2dc6b37d61b4c0
Tianyi Xu
claimed_verified
2026-02-11T11:16:53.605Z
6518a144a28f86d3e9e67c34
/avatars/f2aed39e971cffe6c9d0b9c2f7a0df70.svg
Tianyi Xu
tianyi0216
user
tianyi0216
698b396b1b2dc6b37d61b4c1
Yuzheng Hu
698b396b1b2dc6b37d61b4c2
Jialin Liu
698b396b1b2dc6b37d61b4c3
Guo Chen
698b396b1b2dc6b37d61b4c4
Tianyu Zhang
698b396b1b2dc6b37d61b4c5
Junhao Zheng
698b396b1b2dc6b37d61b4c6
Kexin Yang
698b396b1b2dc6b37d61b4c7
Xingzhang Ren
698b396b1b2dc6b37d61b4c8
Dayiheng Liu
698b396b1b2dc6b37d61b4c9
Linfeng Zhang
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7\% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. In pre-training of GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B tokens, OPUS outperforms industrial-level baselines and even full 200B-token training. Moreover, when combined with industrial-level static filters, OPUS further improves pre-training efficiency, even with lower-quality data. Furthermore, in continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains in specialized domains.
OPUS is a dynamic data selection framework that improves pre-training efficiency by scoring data candidates based on optimizer-induced update projections in a stable proxy-derived target space, achieving superior performance with reduced computational overhead.
data selection optimizer-induced update space effective updates stable in-distribution proxy Ghost technique CountSketch Boltzmann sampling pre-training GPT-2 Qwen3-8B-Base FineWeb FineWeb-Edu SciencePedia