ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
Frontier AI agents show increasing promise as scientific research assistants,
and may eventually be useful for extended, open-ended research workflows.
However, in order to use agents for novel research, we must first assess the
underlying faithfulness and correctness of their work. To evaluate agents as
research assistants, we introduce ReplicationBench, an evaluation framework
that tests whether agents can replicate entire research papers drawn from the
astrophysics literature. Astrophysics, where research relies heavily on
archival data and computational study while requiring little real-world
experimentation, is a particularly useful testbed for AI agents in scientific
research. We split each paper into tasks which require agents to replicate the
paper's core contributions, including the experimental setup, derivations, data
analysis, and codebase. Each task is co-developed with the original paper
authors and targets a key scientific result, enabling objective evaluation of
both faithfulness (adherence to original methods) and correctness (technical
accuracy of results). ReplicationBench is extremely challenging for current
frontier language models: even the best-performing language models score under
20%. We analyze ReplicationBench trajectories in collaboration with domain
experts and find a rich, diverse set of failure modes for agents in scientific
research. ReplicationBench establishes the first benchmark of paper-scale,
expert-validated astrophysics research tasks, reveals insights about agent
performance generalizable to other domains of data-driven science, and provides
a scalable framework for measuring AI agents' reliability in scientific
research.
690172c7646208eac0d1f348
Christine Ye
690172c7646208eac0d1f349
Sihan Yuan
690172c7646208eac0d1f34a
Suchetha Cooray
690172c7646208eac0d1f34b
Steven Dillmann
690172c7646208eac0d1f34c
Ian L. V. Roque
690172c7646208eac0d1f34d
Dalya Baron
690172c7646208eac0d1f34e
Philipp Frank
690172c7646208eac0d1f34f
Sergio Martin-Alvarez
690172c7646208eac0d1f350
Nolan Koblischke
690172c7646208eac0d1f351
Frank J Qu
690172c7646208eac0d1f352
Diyi Yang
690172c7646208eac0d1f353
Risa Wechsler
690172c7646208eac0d1f354
Ioana Ciuca
ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
Frontier AI agents show increasing promise as scientific research assistants,
and may eventually be useful for extended, open-ended research workflows.
However, in order to use agents for novel research, we must first assess the
underlying faithfulness and correctness of their work. To evaluate agents as
research assistants, we introduce ReplicationBench, an evaluation framework
that tests whether agents can replicate entire research papers drawn from the
astrophysics literature. Astrophysics, where research relies heavily on
archival data and computational study while requiring little real-world
experimentation, is a particularly useful testbed for AI agents in scientific
research. We split each paper into tasks which require agents to replicate the
paper's core contributions, including the experimental setup, derivations, data
analysis, and codebase. Each task is co-developed with the original paper
authors and targets a key scientific result, enabling objective evaluation of
both faithfulness (adherence to original methods) and correctness (technical
accuracy of results). ReplicationBench is extremely challenging for current
frontier language models: even the best-performing language models score under
20%. We analyze ReplicationBench trajectories in collaboration with domain
experts and find a rich, diverse set of failure modes for agents in scientific
research. ReplicationBench establishes the first benchmark of paper-scale,
expert-validated astrophysics research tasks, reveals insights about agent
performance generalizable to other domains of data-driven science, and provides
a scalable framework for measuring AI agents' reliability in scientific
research.
ReplicationBench evaluates AI agents' ability to replicate astrophysics research papers, providing insights into their faithfulness and correctness in scientific research tasks.
ReplicationBench AI agents scientific research assistants faithfulness correctness research papers astrophysics experimental setup derivations data analysis codebase language models benchmark expert-validated data-driven science reliability