NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.
6940cff365f1e24a117803e5
Jingzhe Ding
claimed_verified
2025-12-16T10:18:25.834Z
6737eb499ab5a7623aaf038a
/avatars/dfe21c7db767a34f2aff3e80cebcfa22.svg
Jingzhe Ding
JingzheDing
user
JingzheDing
6940cff365f1e24a117803e6
Shengda Long
6940cff365f1e24a117803e7
Changxin Pu
6940cff365f1e24a117803e8
Huan Zhou
6940cff365f1e24a117803e9
Hongwan Gao
6940cff365f1e24a117803ea
Xiang Gao
claimed_verified
2025-12-16T09:47:27.017Z
671068b8243baf4c58fc29c2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/leMkq-kw1icPo2bCLivGv.png
Xiang Gao
coffiney
user
coffiney
6940cff365f1e24a117803eb
Chao He
6940cff365f1e24a117803ec
Yue Hou
claimed_verified
2025-12-16T09:47:24.853Z
694100a19d8967e631cf94cf
/avatars/1c785c0fcd171a62e206ef4074061a81.svg
Yue Hou
YueHou
user
YueHou
6940cff365f1e24a117803ed
Fei Hu
6940cff365f1e24a117803ee
Zhaojian Li
6940cff365f1e24a117803ef
Weiran Shi
6940cff365f1e24a117803f0
Zaiyuan Wang
6940cff365f1e24a117803f1
Daoguang Zan
6940cff365f1e24a117803f2
Chenchen Zhang
6940cff365f1e24a117803f3
Xiaoxu Zhang
6940cff365f1e24a117803f4
Qizhi Chen
6940cff365f1e24a117803f5
Xianfu Cheng
6940cff365f1e24a117803f6
Bo Deng
6940cff365f1e24a117803f7
Qingshui Gu
6940cff365f1e24a117803f8
Kai Hua
claimed_verified
2025-12-16T09:47:28.890Z
64e851825ddcace745ba15bd
/avatars/7b6612c411222974d9ea181784eef915.svg
Kai Hua
kkish
user
kkish
6940cff365f1e24a117803f9
Juntao Lin
admin_assigned
2026-02-10T19:42:59.160Z
6973c033278c2a7caed5a61f
https://cdn-avatars.huggingface.co/v1/production/uploads/6973c033278c2a7caed5a61f/RYkXP9pGAsgYxXi1xwRdK.jpeg
arisumika
Arisumika
user
Arisumika
6940cff365f1e24a117803fa
Pai Liu
6940cff365f1e24a117803fb
Mingchen Li
6940cff365f1e24a117803fc
Xuanguang Pan
6940cff365f1e24a117803fd
Zifan Peng
6940cff365f1e24a117803fe
Yujia Qin
6940cff365f1e24a117803ff
Yong Shan
claimed_verified
2025-12-16T14:10:43.971Z
64b8cc7ebe76d2ff0703bfb3
/avatars/f1c7ff17fd923f1460d362333d9fbfe3.svg
yong
yo37
user
yo37
6940cff365f1e24a11780400
Zhewen Tan
6940cff365f1e24a11780401
Weihao Xie
claimed_verified
2026-02-12T13:59:02.125Z
6657620ea496f7fcb67c3871
/avatars/54fef1c835e6f6b478652d438a140d45.svg
xieweihao
chalengr
user
chalengr
6940cff365f1e24a11780402
Zihan Wang
6940cff365f1e24a11780403
Yishuo Yuan
6940cff365f1e24a11780404
Jiayu Zhang
6940cff365f1e24a11780405
Enduo Zhao
6940cff365f1e24a11780406
Yunfei Zhao
6940cff365f1e24a11780407
He Zhu
6940cff365f1e24a11780408
Chenyang Zou
6940cff365f1e24a11780409
Ming Ding
6940cff365f1e24a1178040a
Jianpeng Jiao
6940cff365f1e24a1178040b
Jiaheng Liu
6940cff365f1e24a1178040c
Minghao Liu
claimed_verified
2026-01-06T10:04:46.803Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
Liam-Liu
user
Liam-Liu
6940cff365f1e24a1178040d
Qian Liu
6940cff365f1e24a1178040e
Chongyao Tao
claimed_verified
2025-12-16T09:47:30.496Z
648db8f92eb481bc8bc60469
/avatars/7a640980ed225e31117f750e09bcc18c.svg
Chongyang Tao
chongyang09
user
chongyang09
6940cff365f1e24a1178040f
Jian Yang
6940cff365f1e24a11780410
Tong Yang
6940cff365f1e24a11780411
Zhaoxiang Zhang
6940cff365f1e24a11780412
Xinjie Chen
6940cff365f1e24a11780413
Wenhao Huang
6940cff365f1e24a11780414
Ge Zhang
claimed_verified
2026-01-12T10:37:24.414Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.
NL2Repo Bench evaluates long-horizon software development capabilities of coding agents by assessing their ability to generate complete Python libraries from natural-language requirements.
NL2Repo Bench long-horizon repository generation autonomous software development coding agents natural-language requirements architecture design dependency management multi-module logic installable Python library test pass rates global coherence cross-file dependencies long-horizon reasoning