EvoClaw: Evaluating AI Agents on Continuous Software Evolution
With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as semantically cohesive development goals. These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
69b8df4f6b3f36c249fbc4e3
Gangda Deng
admin_assigned
2026-03-17T08:50:30.896Z
62c54062f7c810d3cd4043cc
/avatars/bc0be3082fe1a6eec7687f57cfe84d55.svg
Gangda Deng
Hydrapse
user
Hydrapse
69b8df4f6b3f36c249fbc4e4
Zhaoling Chen
69b8df4f6b3f36c249fbc4e5
Zhongming Yu
admin_assigned
2026-03-17T08:50:42.296Z
652622617fa0bebfbf0e5dd1
/avatars/765ad731868431b80ea751d666998714.svg
Zhongming Yu
fishmingyu
user
fishmingyu
69b8df4f6b3f36c249fbc4e6
Haoyang Fan
69b8df4f6b3f36c249fbc4e7
Yuhong Liu
69b8df4f6b3f36c249fbc4e8
Yuxin Yang
69b8df4f6b3f36c249fbc4e9
Dhruv Parikh
69b8df4f6b3f36c249fbc4ea
Rajgopal Kannan
69b8df4f6b3f36c249fbc4eb
Le Cong
69b8df4f6b3f36c249fbc4ec
Mengdi Wang
69b8df4f6b3f36c249fbc4ed
Qian Zhang
69b8df4f6b3f36c249fbc4ee
Viktor Prasanna
69b8df4f6b3f36c249fbc4ef
Xiangru Tang
admin_assigned
2026-03-17T08:51:23.717Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
RTT1
user
RTT1
69b8df4f6b3f36c249fbc4f0
Xingyao Wang
admin_assigned
2026-03-17T08:51:30.078Z
62c63f31031996c36c848504
https://cdn-avatars.huggingface.co/v1/production/uploads/62c63f31031996c36c848504/xZvH6jnozuFLXbtRSU-h1.jpeg
Xingyao Wang
xingyaoww
user
xingyaoww
EvoClaw: Evaluating AI Agents on Continuous Software Evolution
With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as semantically cohesive development goals. These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.