DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.
69ccf51d1f6782218d973dbc
Hao Liang
claimed_verified
2026-04-03T09:08:55.763Z
6751a4fedf636b0140a9b873
/avatars/d75f7f6cfbfb4d646e0e557d1cfacdce.svg
Hao Liang
lhpku20010120
user
lhpku20010120
69ccf51d1f6782218d973dbd
Zhengyang Zhao
69ccf51d1f6782218d973dbe
Meiyi Qiang
69ccf51d1f6782218d973dbf
Mingrui Chen
claimed_verified
2026-04-07T08:46:44.977Z
64049e1d0ab5e22719f37be2
/avatars/1902cd469d231dc252ac92d33af8f5b2.svg
Mingrui Chen
Walnutes
user
Walnutes
69ccf51d1f6782218d973dc0
Lu Ma
69ccf51d1f6782218d973dc1
Rongyi Yu
69ccf51d1f6782218d973dc2
Hengyi Feng
69ccf51d1f6782218d973dc3
Shixuan Sun
69ccf51d1f6782218d973dc4
Zimo Meng
69ccf51d1f6782218d973dc5
Xiaochen Ma
claimed_verified
2026-04-10T07:45:25.046Z
65099d08f37afbab0d3fb268
/avatars/cef45b7c6b7c90bbef341a39a9bb51be.svg
Xiaochen Ma
Sunnyhaze
user
Sunnyhaze
69ccf51d1f6782218d973dc6
Xuanlin Yang
69ccf51d1f6782218d973dc7
Qifeng Cai
69ccf51d1f6782218d973dc8
Ruichuan An
69ccf51d1f6782218d973dc9
Bohan Zeng
claimed_verified
2026-04-07T08:46:42.874Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
zbhpku
user
zbhpku
69ccf51d1f6782218d973dca
Zhen Hao Wong
69ccf51d1f6782218d973dcb
Chengyu Shen
69ccf51d1f6782218d973dcc
Runming He
69ccf51d1f6782218d973dcd
Zhaoyang Han
69ccf51d1f6782218d973dce
Yaowei Zheng
69ccf51d1f6782218d973dcf
Fangcheng Fu
69ccf51d1f6782218d973dd0
Conghui He
69ccf51d1f6782218d973dd1
Bin Cui
69ccf51d1f6782218d973dd2
Zhiyu Li
69ccf51d1f6782218d973dd3
Weinan E
69ccf51d1f6782218d973dd4
Wentao Zhang
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.
DataFlex is a unified framework for dynamic data-centric training of large language models that supports sample selection, domain mixture adjustment, and sample reweighting while maintaining compatibility with standard training workflows and enabling efficient large-scale deployment.
data-centric training large language models sample selection domain mixture adjustment sample reweighting LLaMA-Factory extensible trainer abstractions modular components embedding extraction gradient computation DeepSpeed ZeRO-3 MMLU perplexity Qwen2.5-1.5B SlimPajama