DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI
The rapidly growing demand for high-quality data in Large Language Models (LLMs) has intensified the need for scalable, reliable, and semantically rich data preparation pipelines. However, current practices remain dominated by ad-hoc scripts and loosely specified workflows, which lack principled abstractions, hinder reproducibility, and offer limited support for model-in-the-loop data generation. To address these challenges, we present DataFlow, a unified and extensible LLM-driven data preparation framework. DataFlow is designed with system-level abstractions that enable modular, reusable, and composable data transformations, and provides a PyTorch-style pipeline construction API for building debuggable and optimizable dataflows. The framework consists of nearly 200 reusable operators and six domain-general pipelines spanning text, mathematical reasoning, code, Text-to-SQL, agentic RAG, and large-scale knowledge extraction. To further improve usability, we introduce DataFlow-Agent, which automatically translates natural-language specifications into executable pipelines via operator synthesis, pipeline planning, and iterative verification. Across six representative use cases, DataFlow consistently improves downstream LLM performance. Our math, code, and text pipelines outperform curated human datasets and specialized synthetic baselines, achieving up to +3\% execution accuracy in Text-to-SQL over SynSQL, +7\% average improvements on code benchmarks, and 1--3 point gains on MATH, GSM8K, and AIME. Moreover, a unified 10K-sample dataset produced by DataFlow enables base models to surpass counterparts trained on 1M Infinity-Instruct data. These results demonstrate that DataFlow provides a practical and high-performance substrate for reliable, reproducible, and scalable LLM data preparation, and establishes a system-level foundation for future data-centric AI development.
6949026334f46eaf46cbb3d1
Hao Liang
claimed_verified
2026-04-03T09:09:52.246Z
6751a4fedf636b0140a9b873
/avatars/d75f7f6cfbfb4d646e0e557d1cfacdce.svg
Hao Liang
lhpku20010120
user
lhpku20010120
6949026334f46eaf46cbb3d2
Xiaochen Ma
claimed_verified
2025-12-25T20:51:26.454Z
65099d08f37afbab0d3fb268
/avatars/cef45b7c6b7c90bbef341a39a9bb51be.svg
Xiaochen Ma
Sunnyhaze
user
Sunnyhaze
6949026334f46eaf46cbb3d3
Zhou Liu
6949026334f46eaf46cbb3d4
Zhen Hao Wong
claimed_verified
2025-12-25T20:51:24.004Z
65d3149f0545ab7c11568b2b
/avatars/02f36d8320b8331849e7d7f40e509ca8.svg
Zhen Hao Wong
aaron1141
user
aaron1141
6949026334f46eaf46cbb3d5
Zhengyang Zhao
6949026334f46eaf46cbb3d6
Zimo Meng
6949026334f46eaf46cbb3d7
Runming He
6949026334f46eaf46cbb3d8
Chengyu Shen
6949026334f46eaf46cbb3d9
Qifeng Cai
6949026334f46eaf46cbb3da
Zhaoyang Han
6949026334f46eaf46cbb3db
Meiyi Qiang
6949026334f46eaf46cbb3dc
Yalin Feng
6949026334f46eaf46cbb3dd
Tianyi Bai
6949026334f46eaf46cbb3de
Zewei Pan
claimed_verified
2025-12-29T14:25:06.525Z
658a907a15b65eb9ba2f52e8
/avatars/64575a37a6e449ad525877a70eafa1be.svg
zewei pan
pzp5700
user
pzp5700
6949026334f46eaf46cbb3df
Ziyi Guo
6949026334f46eaf46cbb3e0
Yizhen Jiang
6949026334f46eaf46cbb3e1
Jingwen Deng
6949026334f46eaf46cbb3e2
Qijie You
6949026334f46eaf46cbb3e3
Peichao Lai
6949026334f46eaf46cbb3e4
Tianyu Guo
6949026334f46eaf46cbb3e5
Chi Hsu Tsai
6949026334f46eaf46cbb3e6
Hengyi Feng
6949026334f46eaf46cbb3e7
Rui Hu
6949026334f46eaf46cbb3e8
Wenkai Yu
6949026334f46eaf46cbb3e9
Junbo Niu
6949026334f46eaf46cbb3ea
Bohan Zeng
claimed_verified
2026-04-07T08:49:41.764Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
zbhpku
user
zbhpku
6949026334f46eaf46cbb3eb
Ruichuan An
6949026334f46eaf46cbb3ec
Lu Ma
6949026334f46eaf46cbb3ed
Jihao Huang
6949026334f46eaf46cbb3ee
Yaowei Zheng
claimed_verified
2025-12-31T20:57:58.594Z
642fef28a043f0ac7defa8a9
https://cdn-avatars.huggingface.co/v1/production/uploads/642fef28a043f0ac7defa8a9/RwOEkuj3fOnOA54tGR7Ea.png
Yaowei Zheng
hiyouga
user
hiyouga
6949026334f46eaf46cbb3ef
Conghui He
6949026334f46eaf46cbb3f0
Linpeng Tang
6949026334f46eaf46cbb3f1
Bin Cui
6949026334f46eaf46cbb3f2
Weinan E
6949026334f46eaf46cbb3f3
Wentao Zhang
DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI
The rapidly growing demand for high-quality data in Large Language Models (LLMs) has intensified the need for scalable, reliable, and semantically rich data preparation pipelines. However, current practices remain dominated by ad-hoc scripts and loosely specified workflows, which lack principled abstractions, hinder reproducibility, and offer limited support for model-in-the-loop data generation. To address these challenges, we present DataFlow, a unified and extensible LLM-driven data preparation framework. DataFlow is designed with system-level abstractions that enable modular, reusable, and composable data transformations, and provides a PyTorch-style pipeline construction API for building debuggable and optimizable dataflows. The framework consists of nearly 200 reusable operators and six domain-general pipelines spanning text, mathematical reasoning, code, Text-to-SQL, agentic RAG, and large-scale knowledge extraction. To further improve usability, we introduce DataFlow-Agent, which automatically translates natural-language specifications into executable pipelines via operator synthesis, pipeline planning, and iterative verification. Across six representative use cases, DataFlow consistently improves downstream LLM performance. Our math, code, and text pipelines outperform curated human datasets and specialized synthetic baselines, achieving up to +3\% execution accuracy in Text-to-SQL over SynSQL, +7\% average improvements on code benchmarks, and 1--3 point gains on MATH, GSM8K, and AIME. Moreover, a unified 10K-sample dataset produced by DataFlow enables base models to surpass counterparts trained on 1M Infinity-Instruct data. These results demonstrate that DataFlow provides a practical and high-performance substrate for reliable, reproducible, and scalable LLM data preparation, and establishes a system-level foundation for future data-centric AI development.
DataFlow is an LLM-driven data preparation framework that enhances data quality and reproducibility for various tasks, improving LLM performance with automatically generated pipelines.
DataFlow Large Language Models (LLMs) data preparation pipelines system-level abstractions PyTorch-style pipeline construction API reusable operators domain-general pipelines Text-to-SQL agentic RAG large-scale knowledge extraction DataFlow-Agent operator synthesis pipeline planning iterative verification