AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform near or below the median of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .
69c0b71c3fd3fef6237a6873
An Luo
claimed_verified
2026-03-24T09:22:58.099Z
65cf12f7d8b82d378f816e14
/avatars/c5e6d9d823f9036ca5645b2f7b7bbb27.svg
An Luo
lainmn
user
lainmn
69c0b71c3fd3fef6237a6874
Jin Du
69c0b71c3fd3fef6237a6875
Xun Xian
69c0b71c3fd3fef6237a6876
Robert Specht
69c0b71c3fd3fef6237a6877
Fangqiao Tian
69c0b71c3fd3fef6237a6878
Ganghua Wang
69c0b71c3fd3fef6237a6879
Xuan Bi
69c0b71c3fd3fef6237a687a
Charles Fleming
69c0b71c3fd3fef6237a687b
Ashish Kundu
69c0b71c3fd3fef6237a687c
Jayanth Srinivasa
69c0b71c3fd3fef6237a687d
Mingyi Hong
69c0b71c3fd3fef6237a687e
Rui Zhang
69c0b71c3fd3fef6237a687f
Tianxi Li
69c0b71c3fd3fef6237a6880
Galin Jones
69c0b71c3fd3fef6237a6881
Jie Ding
AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform near or below the median of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .
AgentDS benchmark evaluates AI agents and human-AI collaboration in domain-specific data science tasks, revealing continued necessity of human expertise despite advances in large language models and AI agents.
large language models artificial intelligence agents data science workflow domain-specific reasoning human-AI collaboration benchmark competition