BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
Current benchmarks for code agents primarily assess narrow, repository-specific fixes, overlooking critical real-world challenges such as cross-repository reasoning, domain-specialized problem solving, dependency-driven migration, and full-repository generation. To address this gap, we introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope - using 500 real-world instances across four distinct settings. Experimental results reveal a significant capability gap: even frontier models plateau below 45% success, and no single model performs consistently across task types. To systematically investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities. Our experiments show that search augmentation yields inconsistent gains and can in some cases degrade performance, highlighting the difficulty of emulating developer-like workflows that interleave search and reasoning during coding tasks. This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
69a7a7c52833ff1122ff6472
Guoxin Chen
69a7a7c52833ff1122ff6473
Fanzhe Meng
69a7a7c52833ff1122ff6474
Jiale Zhao
69a7a7c52833ff1122ff6475
Minghao Li
69a7a7c52833ff1122ff6476
Daixuan Cheng
claimed_verified
2026-03-04T09:22:39.129Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
daixuancheng
user
daixuancheng
69a7a7c52833ff1122ff6477
Huatong Song
69a7a7c52833ff1122ff6478
Jie Chen
69a7a7c52833ff1122ff6479
Yuzhi Lin
69a7a7c52833ff1122ff647a
Hui Chen
69a7a7c52833ff1122ff647b
Xin Zhao
69a7a7c52833ff1122ff647c
Ruihua Song
69a7a7c52833ff1122ff647d
Chang Liu
69a7a7c52833ff1122ff647e
Cheng Chen
69a7a7c52833ff1122ff647f
Kai Jia
69a7a7c52833ff1122ff6480
Ji-Rong Wen
BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
Current benchmarks for code agents primarily assess narrow, repository-specific fixes, overlooking critical real-world challenges such as cross-repository reasoning, domain-specialized problem solving, dependency-driven migration, and full-repository generation. To address this gap, we introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope - using 500 real-world instances across four distinct settings. Experimental results reveal a significant capability gap: even frontier models plateau below 45% success, and no single model performs consistently across task types. To systematically investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities. Our experiments show that search augmentation yields inconsistent gains and can in some cases degrade performance, highlighting the difficulty of emulating developer-like workflows that interleave search and reasoning during coding tasks. This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
Current code agent benchmarks fail to capture real-world complexity, prompting the creation of BeyondSWE to evaluate broader reasoning and knowledge scopes, alongside SearchSWE to study external knowledge integration in coding tasks.
code agents benchmarks cross-repository reasoning domain-specialized problem solving dependency-driven migration full-repository generation SearchSWE deep search coding abilities external knowledge developer-like workflows reasoning performance evaluation