AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research
  Suite
AI agents hold the potential to revolutionize scientific productivity by
automating literature reviews, replicating experiments, analyzing data, and
even proposing new directions of inquiry; indeed, there are now many such
agents, ranging from general-purpose "deep research" systems to specialized
science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of
these agents is critical for progress. Yet existing benchmarks fall short on
several fronts: they (1) fail to provide holistic, product-informed measures of
real-world use cases such as science research; (2) lack reproducible agent
tools necessary for a controlled comparison of core agentic capabilities; (3)
do not account for confounding variables such as model cost and tool access;
(4) do not provide standardized interfaces for quick agent prototyping and
evaluation; and (5) lack comprehensive baseline agents necessary to identify
true advances. In response, we define principles and tooling for more
rigorously benchmarking agents. Using these, we present AstaBench, a suite that
provides the first holistic measure of agentic ability to perform scientific
research, comprising 2400+ problems spanning the entire scientific discovery
process and multiple scientific domains, and including many problems inspired
by actual user requests to deployed Asta agents. Our suite comes with the first
scientific research environment with production-grade search tools that enable
controlled, reproducible evaluation, better accounting for confounders.
Alongside, we provide a comprehensive suite of nine science-optimized classes
of Asta agents and numerous baselines. Our extensive evaluation of 57 agents
across 22 agent classes reveals several interesting findings, most importantly
that despite meaningful progress on certain individual aspects, AI remains far
from solving the challenge of science research assistance.
68fed3ee6cdff8b857f47128
Jonathan Bragg
68fed3ee6cdff8b857f47129
Mike D'Arcy
68fed3ee6cdff8b857f4712a
Nishant Balepur
68fed3ee6cdff8b857f4712b
Dan Bareket
68fed3ee6cdff8b857f4712c
Bhavana Dalvi
68fed3ee6cdff8b857f4712d
Sergey Feldman
68fed3ee6cdff8b857f4712e
Dany Haddad
68fed3ee6cdff8b857f4712f
Jena D. Hwang
68fed3ee6cdff8b857f47130
Peter Jansen
68fed3ee6cdff8b857f47131
Varsha Kishore
68fed3ee6cdff8b857f47132
Bodhisattwa Prasad Majumder
68fed3ee6cdff8b857f47133
Aakanksha Naik
68fed3ee6cdff8b857f47134
Sigal Rahamimov
68fed3ee6cdff8b857f47135
Kyle Richardson
68fed3ee6cdff8b857f47136
Amanpreet Singh
claimed_verified
2026-03-09T11:06:04.481Z
633b34009fe04b13f46be3e8
/avatars/81a02528351c5583373fa5b96840e99d.svg
Amanpreet Singh
aps6992
user
aps6992
68fed3ee6cdff8b857f47137
Harshit Surana
68fed3ee6cdff8b857f47138
Aryeh Tiktinsky
68fed3ee6cdff8b857f47139
Rosni Vasu
68fed3ee6cdff8b857f4713a
Guy Wiener
68fed3ee6cdff8b857f4713b
Chloe Anastasiades
68fed3ee6cdff8b857f4713c
Stefan Candra
68fed3ee6cdff8b857f4713d
Jason Dunkelberger
68fed3ee6cdff8b857f4713e
Dan Emery
68fed3ee6cdff8b857f4713f
Rob Evans
68fed3ee6cdff8b857f47140
Malachi Hamada
68fed3ee6cdff8b857f47141
Regan Huff
68fed3ee6cdff8b857f47142
Rodney Kinney
68fed3ee6cdff8b857f47143
Matt Latzke
68fed3ee6cdff8b857f47144
Jaron Lochner
68fed3ee6cdff8b857f47145
Ruben Lozano-Aguilera
68fed3ee6cdff8b857f47146
Cecile Nguyen
68fed3ee6cdff8b857f47147
Smita Rao
68fed3ee6cdff8b857f47148
Amber Tanaka
68fed3ee6cdff8b857f47149
Brooke Vlahos
68fed3ee6cdff8b857f4714a
Peter Clark
68fed3ee6cdff8b857f4714b
Doug Downey
68fed3ee6cdff8b857f4714c
Yoav Goldberg
68fed3ee6cdff8b857f4714d
Ashish Sabharwal
68fed3ee6cdff8b857f4714e
Daniel S. Weld
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research
  Suite
AI agents hold the potential to revolutionize scientific productivity by
automating literature reviews, replicating experiments, analyzing data, and
even proposing new directions of inquiry; indeed, there are now many such
agents, ranging from general-purpose "deep research" systems to specialized
science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of
these agents is critical for progress. Yet existing benchmarks fall short on
several fronts: they (1) fail to provide holistic, product-informed measures of
real-world use cases such as science research; (2) lack reproducible agent
tools necessary for a controlled comparison of core agentic capabilities; (3)
do not account for confounding variables such as model cost and tool access;
(4) do not provide standardized interfaces for quick agent prototyping and
evaluation; and (5) lack comprehensive baseline agents necessary to identify
true advances. In response, we define principles and tooling for more
rigorously benchmarking agents. Using these, we present AstaBench, a suite that
provides the first holistic measure of agentic ability to perform scientific
research, comprising 2400+ problems spanning the entire scientific discovery
process and multiple scientific domains, and including many problems inspired
by actual user requests to deployed Asta agents. Our suite comes with the first
scientific research environment with production-grade search tools that enable
controlled, reproducible evaluation, better accounting for confounders.
Alongside, we provide a comprehensive suite of nine science-optimized classes
of Asta agents and numerous baselines. Our extensive evaluation of 57 agents
across 22 agent classes reveals several interesting findings, most importantly
that despite meaningful progress on certain individual aspects, AI remains far
from solving the challenge of science research assistance.
AstaBench provides a comprehensive benchmark suite for evaluating AI agents in scientific research, revealing that while progress has been made, AI still falls short in fully assisting scientific research.
AI agents deep research systems AI Scientist AIGS AstaBench scientific discovery process production-grade search tools science-optimized agents baseline agents