Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such as math and code. However, identifying an optimal mixture remains an open challenge, as existing approaches either rely on unreliable tiny-scale proxy experiments or require prohibitively expensive large-scale exploration. To address this, we propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios. Instead of training proxy models for every sampled mixture, DeMix trains component models on candidate datasets at scale and derives data mixture proxies via weighted model merging. This paradigm decouples search from training costs, enabling evaluation of unlimited sampled mixtures without extra training burden and thus facilitating better mixture discovery through more search trials. Extensive experiments demonstrate that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost. Additionally, we release the DeMix Corpora, a comprehensive 22T-token dataset comprising high-quality pre-training data with validated mixtures to facilitate open research. Our code and DeMix Corpora is available at https://github.com/Lucius-lsr/DeMix.
6981faf447987be58cdb7d13
Shengrui Li
6981faf447987be58cdb7d14
Fei Zhao
claimed_verified
2026-02-04T12:31:37.433Z
65328aa39326d6da5ff19b52
/avatars/5c3de984cd6eba69616bb608796865c5.svg
Fei Zhao
Hiiamein
user
Hiiamein
6981faf447987be58cdb7d15
Kaiyan Zhao
6981faf447987be58cdb7d16
Jieying Ye
6981faf447987be58cdb7d17
Haifeng Liu
6981faf447987be58cdb7d18
Fangcheng Shi
6981faf447987be58cdb7d19
Zheyong Xie
6981faf447987be58cdb7d1a
Yao Hu
6981faf447987be58cdb7d1b
Shaosheng Cao
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such as math and code. However, identifying an optimal mixture remains an open challenge, as existing approaches either rely on unreliable tiny-scale proxy experiments or require prohibitively expensive large-scale exploration. To address this, we propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios. Instead of training proxy models for every sampled mixture, DeMix trains component models on candidate datasets at scale and derives data mixture proxies via weighted model merging. This paradigm decouples search from training costs, enabling evaluation of unlimited sampled mixtures without extra training burden and thus facilitating better mixture discovery through more search trials. Extensive experiments demonstrate that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost. Additionally, we release the DeMix Corpora, a comprehensive 22T-token dataset comprising high-quality pre-training data with validated mixtures to facilitate open research. Our code and DeMix Corpora is available at https://github.com/Lucius-lsr/DeMix.
DeMix is a framework that uses model merging to predict optimal data ratios for LLM pre-training, decoupling search from training costs to improve mixture discovery efficiency.
Large Language Model pre-training data mixture model merging weighted model merging proxy experiments training costs search trials benchmark performance