PRBench: End-to-end Paper Reproduction in Physics Research
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
69cb2cff22f9e9dce3fe1e9b
Shi Qiu
69cb2cff22f9e9dce3fe1e9c
Junyi Deng
69cb2cff22f9e9dce3fe1e9d
Yiwei Deng
69cb2cff22f9e9dce3fe1e9e
Haoran Dong
69cb2cff22f9e9dce3fe1e9f
Jieyu Fu
69cb2cff22f9e9dce3fe1ea0
Mao Li
69cb2cff22f9e9dce3fe1ea1
Zeyu Li
69cb2cff22f9e9dce3fe1ea2
Zhaolong Zhang
69cb2cff22f9e9dce3fe1ea3
Huiwen Zheng
69cb2cff22f9e9dce3fe1ea4
Leidong Bao
claimed_verified
2026-03-31T09:28:50.570Z
69cb5cb1a029f6c70f1708c0
/avatars/c0f2c5a36d55947d9ca5efdf565b5bf8.svg
Bao Leidong
Muon227
user
Muon227
69cb2cff22f9e9dce3fe1ea5
Anqi Lv
69cb2cff22f9e9dce3fe1ea6
Zihan Mo
69cb2cff22f9e9dce3fe1ea7
Yadi Niu
69cb2cff22f9e9dce3fe1ea8
Yiyang Peng
69cb2cff22f9e9dce3fe1ea9
Yu Tian
69cb2cff22f9e9dce3fe1eaa
Yili Wang
69cb2cff22f9e9dce3fe1eab
Ziyu Wang
69cb2cff22f9e9dce3fe1eac
Zi-Yu Wang
69cb2cff22f9e9dce3fe1ead
Jiashen Wei
69cb2cff22f9e9dce3fe1eae
Liuheng Wu
69cb2cff22f9e9dce3fe1eaf
Aoran Xue
69cb2cff22f9e9dce3fe1eb0
Leyi Yang
69cb2cff22f9e9dce3fe1eb1
Guanglu Yuan
69cb2cff22f9e9dce3fe1eb2
Xiarui Zhan
69cb2cff22f9e9dce3fe1eb3
Jingjun Zhang
69cb2cff22f9e9dce3fe1eb4
Zifan Zheng
69cb2cff22f9e9dce3fe1eb5
Pengfei Liu
69cb2cff22f9e9dce3fe1eb6
Linrui Zhen
69cb2cff22f9e9dce3fe1eb7
Kaiyang Li
69cb2cff22f9e9dce3fe1eb8
Qichang Li
69cb2cff22f9e9dce3fe1eb9
Ziheng Zhou
69cb2cff22f9e9dce3fe1eba
Guo-En Nian
69cb2cff22f9e9dce3fe1ebb
Yunwei Xiao
69cb2cff22f9e9dce3fe1ebc
Qing-Hong Cao
69cb2cff22f9e9dce3fe1ebd
Linjie Dai
69cb2cff22f9e9dce3fe1ebe
Xu Feng
69cb2cff22f9e9dce3fe1ebf
Peng Gao
69cb2cff22f9e9dce3fe1ec0
Ying Gu
69cb2cff22f9e9dce3fe1ec1
Chang Liu
69cb2cff22f9e9dce3fe1ec2
Jia Liu
69cb2cff22f9e9dce3fe1ec3
Ming-xing Luo
69cb2cff22f9e9dce3fe1ec4
Yan-Qing Ma
69cb2cff22f9e9dce3fe1ec5
Liang-You Peng
69cb2cff22f9e9dce3fe1ec6
Huichao Song
69cb2cff22f9e9dce3fe1ec7
Shufeng Wang
69cb2cff22f9e9dce3fe1ec8
Chenxu Wang
69cb2cff22f9e9dce3fe1ec9
Tao Wang
69cb2cff22f9e9dce3fe1eca
Yi-Nan Wang
69cb2cff22f9e9dce3fe1ecb
Chengyin Wu
69cb2cff22f9e9dce3fe1ecc
Pengwei Zhao
69cb2cff22f9e9dce3fe1ecd
Hua Xing Zhu
PRBench: End-to-end Paper Reproduction in Physics Research
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
PRBench evaluates AI agents' ability to reproduce scientific research by requiring them to implement algorithms from published papers and match original results, revealing significant challenges in formula implementation, debugging, and data accuracy.
large language models scientific reasoning end-to-end reproduction formula derivation code generation benchmark agentified assessment pipeline coding agents quantitative results numerical simulations