FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.
698d4c7965c0d15a6d1621a7
Qixing Zhou
698d4c7965c0d15a6d1621a8
Jiacheng Zhang
claimed_verified
2026-02-12T13:28:04.846Z
6868f58a4757672a6da7c417
/avatars/73154b7e0f1af68054b97f10a6c2e670.svg
JiaCheng Zhang
jiachengzhg
user
jiachengzhg
698d4c7965c0d15a6d1621a9
Haiyang Wang
claimed_verified
2026-02-12T13:28:02.525Z
65f43c3cc9940817caaf4434
/avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
Haiyang Wang
Haiyang-W
user
Haiyang-W
698d4c7965c0d15a6d1621aa
Rui Hao
698d4c7965c0d15a6d1621ab
Jiahe Wang
698d4c7965c0d15a6d1621ac
Minghao Han
698d4c7965c0d15a6d1621ad
Yuxue Yang
698d4c7965c0d15a6d1621ae
Shuzhe Wu
698d4c7965c0d15a6d1621af
Feiyang Pan
698d4c7965c0d15a6d1621b0
Lue Fan
698d4c7965c0d15a6d1621b1
Dandan Tu
698d4c7965c0d15a6d1621b2
Zhaoxiang Zhang
FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.
FeatureBench evaluates agentic coding performance in comprehensive feature-oriented development through execution-based assessments and automated task derivation from code repositories.
large language models agentic coding software development feature-oriented development execution-based evaluation test-driven method dependency graph unit tests automated task collection data leakage agent training