AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
698b0ed21b2dc6b37d61b3d0
Alisia Lupidi
698b0ed21b2dc6b37d61b3d1
Bhavul Gauri
claimed_verified
2026-02-11T11:17:30.748Z
60720704227ff331937110f4
/avatars/8010bfb98256c138049aa3d237737b37.svg
Bhavul Gauri
bhavul
user
bhavul
698b0ed21b2dc6b37d61b3d2
Thomas Simon Foster
698b0ed21b2dc6b37d61b3d3
Bassel Al Omari
698b0ed21b2dc6b37d61b3d4
Despoina Magka
claimed_verified
2026-04-13T19:26:28.222Z
673dc2345b07d224bff10173
/avatars/2b7b1491de77bc3193fd912de76e176d.svg
Despoina Magka
marlamagka
user
marlamagka
698b0ed21b2dc6b37d61b3d5
Alberto Pepe
claimed_verified
2026-02-11T12:34:25.082Z
6932cd2194556b358ec2c640
/avatars/da9703be57b9c62c4737b591b7695bd1.svg
Alberto Pepe
albertompepe
user
albertompepe
698b0ed21b2dc6b37d61b3d6
Alexis Audran-Reiss
698b0ed21b2dc6b37d61b3d7
Muna Aghamelu
698b0ed21b2dc6b37d61b3d8
Nicolas Baldwin
698b0ed21b2dc6b37d61b3d9
Lucia Cipolina-Kun
698b0ed21b2dc6b37d61b3da
Jean-Christophe Gagnon-Audet
698b0ed21b2dc6b37d61b3db
Chee Hau Leow
698b0ed21b2dc6b37d61b3dc
Sandra Lefdal
698b0ed21b2dc6b37d61b3dd
Hossam Mossalam
698b0ed21b2dc6b37d61b3de
Abhinav Moudgil
claimed_verified
2026-04-01T12:46:36.419Z
6392c2fe388768b9d90d44db
/avatars/a224e14049f814037661388418e9cd13.svg
Abhinav Moudgil
amoudgl
user
amoudgl
698b0ed21b2dc6b37d61b3df
Saba Nazir
claimed_verified
2026-02-11T11:17:23.767Z
697748f611a42e0efd3bddcc
/avatars/d591182b0e55f125691f6031ba395241.svg
Saba Nazir
snazir
user
snazir
698b0ed21b2dc6b37d61b3e0
Emanuel Tewolde
698b0ed21b2dc6b37d61b3e1
Isabel Urrego
698b0ed21b2dc6b37d61b3e2
Jordi Armengol Estape
698b0ed21b2dc6b37d61b3e3
Amar Budhiraja
claimed_verified
2026-02-11T22:16:57.977Z
6687ee79eee600e418404cc9
/avatars/d73d3a360ab3b3ce353a6306c7270a13.svg
Amar Budhiraja
ambud26
user
ambud26
698b0ed21b2dc6b37d61b3e4
Gaurav Chaurasia
claimed_verified
2026-02-26T10:03:58.304Z
691c6b9b660c15d270b5838a
/avatars/2dc40e079bd8b7af7ae4a4ac43acc552.svg
Gaurav Chaurasia
gchauras
user
gchauras
698b0ed21b2dc6b37d61b3e5
Abhishek Charnalia
698b0ed21b2dc6b37d61b3e6
Derek Dunfield
698b0ed21b2dc6b37d61b3e7
Karen Hambardzumyan
claimed_verified
2026-02-11T11:17:21.100Z
63a2fb1cdca41424cb8956bd
https://cdn-avatars.huggingface.co/v1/production/uploads/63a2fb1cdca41424cb8956bd/9BDkkHXPifN-_czXBAecF.png
Karen Hambardzumyan
mahnerak
user
mahnerak
698b0ed21b2dc6b37d61b3e8
Daniel Izcovich
698b0ed21b2dc6b37d61b3e9
Martin Josifoski
698b0ed21b2dc6b37d61b3ea
Ishita Mediratta
claimed_verified
2026-02-11T11:17:28.660Z
67f2839f0d449a5c10d4b67a
/avatars/9fd9b7abb9da1ef68eec461d703cac77.svg
Ish Med
ishmed
user
ishmed
698b0ed21b2dc6b37d61b3eb
Kelvin Niu
698b0ed21b2dc6b37d61b3ec
Parth Pathak
claimed_verified
2026-02-11T11:17:26.106Z
68d66f1a49f157c15451b3c1
/avatars/4ea87b124e07646941b55c6a338561d7.svg
Pathak
papatha
user
papatha
698b0ed21b2dc6b37d61b3ed
Michael Shvartsman
698b0ed21b2dc6b37d61b3ee
Edan Toledo
698b0ed21b2dc6b37d61b3ef
Anton Protopopov
claimed_verified
2026-02-12T20:23:30.082Z
67f00bf17530c3fccbb26c79
/avatars/f0d56f04b1def33dce872a8de71f560d.svg
Anton Protopopov
aprotopopov
user
aprotopopov
698b0ed21b2dc6b37d61b3f0
Roberta Raileanu
698b0ed21b2dc6b37d61b3f1
Alexander Miller
698b0ed21b2dc6b37d61b3f2
Tatiana Shavrina
698b0ed21b2dc6b37d61b3f3
Jakob Foerster
698b0ed21b2dc6b37d61b3f4
Yoram Bachrach
AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
AIRS-Bench presents a comprehensive benchmark suite for evaluating LLM agents across diverse scientific domains, demonstrating current limitations while providing open-source resources for advancement.
LLM agents AI Research Science Benchmark agentic capabilities research lifecycle sequential scaffolds parallel scaffolds human SOTA theoretical performance ceiling