Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents
Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem -- e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context -- but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark (DAB), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB, the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench.
69c419f14ae7eef815b9d886
Ruiying Ma
69c419f14ae7eef815b9d887
Shreya Shankar
69c419f14ae7eef815b9d888
Ruiqi Chen
69c419f14ae7eef815b9d889
Yiming Lin
69c419f14ae7eef815b9d88a
Sepanta Zeighami
69c419f14ae7eef815b9d88b
Rajoshi Ghosh
69c419f14ae7eef815b9d88c
Abhinav Gupta
69c419f14ae7eef815b9d88d
Anushrut Gupta
69c419f14ae7eef815b9d88e
Tanmai Gopal
69c419f14ae7eef815b9d88f
Aditya G. Parameswaran
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents
Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem -- e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context -- but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark (DAB), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB, the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench.
A comprehensive benchmark evaluates enterprise data agents' ability to integrate and analyze multi-database data through natural language, revealing significant challenges in real-world applications.
Data Agent Benchmark natural language queries SQL queries multi-database integration enterprise data agents frontier LLMs pass@1 accuracy