LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~qiu2025locobench assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce LoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.
691d2aae0247f3d258ef3193
Jielin Qiu
691d2aae0247f3d258ef3194
Zuxin Liu
691d2aae0247f3d258ef3195
Zhiwei Liu
691d2aae0247f3d258ef3196
Rithesh Murthy
691d2aae0247f3d258ef3197
Jianguo Zhang
691d2aae0247f3d258ef3198
Haolin Chen
691d2aae0247f3d258ef3199
Shiyu Wang
691d2aae0247f3d258ef319a
Ming Zhu
691d2aae0247f3d258ef319b
Liangwei Yang
691d2aae0247f3d258ef319c
Juntao Tan
691d2aae0247f3d258ef319d
Roshan Ram
691d2aae0247f3d258ef319e
Akshara Prabhakar
691d2aae0247f3d258ef319f
Tulika Awalgaonkar
691d2aae0247f3d258ef31a0
Zixiang Chen
691d2aae0247f3d258ef31a1
Zhepeng Cen
691d2aae0247f3d258ef31a2
Cheng Qian
691d2aae0247f3d258ef31a3
Shelby Heinecke
691d2aae0247f3d258ef31a4
Weiran Yao
691d2aae0247f3d258ef31a5
Silvio Savarese
691d2aae0247f3d258ef31a6
Caiming Xiong
691d2aae0247f3d258ef31a7
Huan Wang
LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~qiu2025locobench assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce LoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.
LoCoBench-Agent evaluates large language models as autonomous software development agents using interactive scenarios, specialized tools, and multi-turn conversations to assess their long-context performance, comprehension, and efficiency.
large language models autonomous agents software development tasks LoCoBench-Agent interactive agent environments multi-turn conversations tool usage efficiency error recovery architectural consistency long-context performance context lengths comprehension-efficiency trade-off conversation efficiency