LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.
699cea804e37ec6dfa1bc476
Yukang Feng
699cea804e37ec6dfa1bc477
Jianwen Sun
699cea804e37ec6dfa1bc478
Zelai Yang
699cea804e37ec6dfa1bc479
Jiaxin Ai
699cea804e37ec6dfa1bc47a
Chuanhao Li
claimed_verified
2026-03-27T11:38:39.321Z
674ea59a8f2e7614a6c72f26
/avatars/86fd4c6d7d33435de49b35659bf65265.svg
Chuanhao
ChuanhaoLi
user
ChuanhaoLi
699cea804e37ec6dfa1bc47b
Zizhen Li
699cea804e37ec6dfa1bc47c
Fanrui Zhang
699cea804e37ec6dfa1bc47d
Kang He
699cea804e37ec6dfa1bc47e
Rui Ma
699cea804e37ec6dfa1bc47f
Jifan Lin
699cea804e37ec6dfa1bc480
Jie Sun
699cea804e37ec6dfa1bc481
Yang Xiao
699cea804e37ec6dfa1bc482
Sizhuo Zhou
699cea804e37ec6dfa1bc483
Wenxiao Wu
699cea804e37ec6dfa1bc484
Yiming Liu
699cea804e37ec6dfa1bc485
Pengfei Liu
699cea804e37ec6dfa1bc486
Yu Qiao
699cea804e37ec6dfa1bc487
Shenglin Zhang
699cea804e37ec6dfa1bc488
Kaipeng Zhang
claimed_verified
2026-03-25T10:51:59.126Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
kpzhang996
user
kpzhang996
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.
LongCLI-Bench evaluates AI agents' ability to complete complex, multi-step programming tasks through command-line interfaces with detailed failure analysis and human-agent collaboration insights.
AI-assisted programming command-line interfaces long-horizon planning agent evaluation requirement fulfillment regression avoidance step-level scoring self-correction human-agent collaboration plan injection interactive guidance