DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.
6976d5d25d41524304c135db
Fan Nie
claimed_verified
2026-01-27T09:43:37.366Z
652fb9e437fad005c70d66f4
https://cdn-avatars.huggingface.co/v1/production/uploads/652fb9e437fad005c70d66f4/xjvro1bFvQ8ILR0tkD6qU.jpeg
Fan Nie
FanNie1208
user
FanNie1208
6976d5d25d41524304c135dc
Junlin Wang
6976d5d25d41524304c135dd
Harper Hua
6976d5d25d41524304c135de
Federico Bianchi
6976d5d25d41524304c135df
Yongchan Kwon
6976d5d25d41524304c135e0
Zhenting Qi
6976d5d25d41524304c135e1
Owen Queen
6976d5d25d41524304c135e2
Shang Zhu
6976d5d25d41524304c135e3
James Zou
DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.
DSGym presents a standardized framework for evaluating data science agents with comprehensive task suites and execution-verified training capabilities.
data science agents DSGym DSGym-Tasks DSBio DSPredict execution-verified data synthesis agent training benchmark evaluation