DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io
693245c66d1060ca587a265c
Fangyu Lei
693245c66d1060ca587a265d
Jinxiang Meng
claimed_verified
2025-12-05T08:39:10.222Z
67f231b5ac0b61b184e84482
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qJZfkOZEn5Zx_VP2MR7ab.png
mengjinxiang
Mjx0221
user
Mjx0221
693245c66d1060ca587a265e
Yiming Huang
693245c66d1060ca587a265f
Junjie Zhao
693245c66d1060ca587a2660
Yitong Zhang
693245c66d1060ca587a2661
Jianwen Luo
claimed_verified
2025-12-05T08:29:34.047Z
66adf5cc0c6056d9f4dc308f
https://cdn-avatars.huggingface.co/v1/production/uploads/66adf5cc0c6056d9f4dc308f/mVKo06P7M1qf6RYNG-c2i.jpeg
Jane Luo
Luo2003
user
Luo2003
693245c66d1060ca587a2662
Xin Zou
693245c66d1060ca587a2663
Ruiyi Yang
693245c66d1060ca587a2664
Wenbo Shi
693245c66d1060ca587a2665
Yan Gao
693245c66d1060ca587a2666
Shizhu He
693245c66d1060ca587a2667
Zuo Wang
693245c66d1060ca587a2668
Qian Liu
693245c66d1060ca587a2669
Yang Wang
693245c66d1060ca587a266a
Ke Wang
693245c66d1060ca587a266b
Jun Zhao
693245c66d1060ca587a266c
Kang Liu
DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io
DAComp is a benchmark of 210 tasks that evaluates the capabilities of agents in real-world data engineering and data analysis workflows, revealing significant deficiencies in both areas.
data engineering data analysis DE tasks DA tasks SQL pipelines multi-metric evaluation LLM-judge hierarchical rubrics autonomous data agents