FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights
Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.
6982c7529084cb4f0ecb576d
Zhen Wang
claimed_verified
2026-02-04T12:28:14.277Z
64a833d2f152bba4b550c913
/avatars/cff37a427c01c6b6691f588481d96416.svg
Zhen Wang
zhenwang9102
user
zhenwang9102
6982c7529084cb4f0ecb576e
Fan Bai
6982c7529084cb4f0ecb576f
Zhongyan Luo
claimed_verified
2026-02-24T09:52:03.052Z
68ad57dd300edba40d7ee760
/avatars/a0f8ef11eb43c375d8c8a7389a6aec93.svg
Zhongyan Luo
silence-suzuki
user
silence-suzuki
6982c7529084cb4f0ecb5770
Jinyan Su
6982c7529084cb4f0ecb5771
Kaiser Sun
claimed_verified
2026-02-04T12:28:09.745Z
616745f893f6837511d9ae25
https://cdn-avatars.huggingface.co/v1/production/uploads/616745f893f6837511d9ae25/H0JIGDL5VOu29zVd4KiKB.jpeg
Kaiser Sun
KaiserWhoLearns
user
KaiserWhoLearns
6982c7529084cb4f0ecb5772
Xinle Yu
6982c7529084cb4f0ecb5773
Jieyuan Liu
6982c7529084cb4f0ecb5774
Kun Zhou
6982c7529084cb4f0ecb5775
Claire Cardie
6982c7529084cb4f0ecb5776
Mark Dredze
6982c7529084cb4f0ecb5777
Eric P. Xing
6982c7529084cb4f0ecb5778
Zhiting Hu
FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights
Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.
Researchers developed FIRE-Bench, a comprehensive evaluation framework that challenges autonomous agents to rediscover established scientific findings through complete research cycles involving hypothesis generation, experimentation, coding, and evidence-based conclusion drawing.
large language models autonomous agents scientific discovery research outputs machine learning research high-impact research empirical evidence experimental design evidence-based reasoning agent systems frontier LLMs research question