AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in Automated Code Review (ACR). However, existing benchmarks suffer from two critical limitations: first, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results; second, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection. To address these challenges, we introduce AACR-Bench a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an "AI-assisted, Expert-verified" annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285% increase in defect coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at https://github.com/alibaba/aacr-bench .
6979c82bdf44b75fa47e48b5
Lei Zhang
claimed_verified
2026-01-30T09:37:23.560Z
6621e0b9761807483a52b1e8
/avatars/4000aeacc49a0fe6b2e36461b0a075d9.svg
Everlearner Zhang
EVZ
user
EVZ
6979c82bdf44b75fa47e48b6
Yongda Yu
admin_assigned
2026-01-28T16:31:02.842Z
647b58bb4c69aa62123a402a
/avatars/efa0f265a7491171d078c5c9701ffd6c.svg
Yongda Yu
icyink
user
icyink
6979c82bdf44b75fa47e48b7
Minghui Yu
claimed_verified
2026-02-02T17:00:47.088Z
69801239b6a2a2f76fa73e00
/avatars/3fed7a992f2a8584b2980688710572b0.svg
Minghui Yu
yuminghui
user
yuminghui
6979c82bdf44b75fa47e48b8
Xinxin Guo
6979c82bdf44b75fa47e48b9
Zhengqi Zhuang
6979c82bdf44b75fa47e48ba
Guoping Rong
6979c82bdf44b75fa47e48bb
Dong Shao
6979c82bdf44b75fa47e48bc
Haifeng Shen
6979c82bdf44b75fa47e48bd
Hongyu Kuang
6979c82bdf44b75fa47e48be
Zhengfeng Li
claimed_verified
2026-01-29T09:17:04.666Z
6969eaf86565de7d93eee247
/avatars/1a65baa0d0d403af927be19f61bb000a.svg
lizhengfeng
kite-lizhengfeng
user
kite-lizhengfeng
6979c82bdf44b75fa47e48bf
Boge Wang
claimed_verified
2026-02-02T17:00:49.067Z
64659821fafbb0cdb40ac713
/avatars/69f43dafaad7820c53a48e030fd9cf75.svg
boge wang
yugixx
user
yugixx
6979c82bdf44b75fa47e48c0
Guoan Zhang
6979c82bdf44b75fa47e48c1
Bangyu Xiang
697c147ba67238fac88cc058
Xiaobin Xu
AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in Automated Code Review (ACR). However, existing benchmarks suffer from two critical limitations: first, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results; second, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection. To address these challenges, we introduce AACR-Bench a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an "AI-assisted, Expert-verified" annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285% increase in defect coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at https://github.com/alibaba/aacr-bench .
AACR-Bench addresses limitations in automated code review benchmarks by providing multi-language, cross-file context with expert-verified defect annotations, revealing significant gaps in prior LLM evaluation and demonstrating that context granularity and retrieval methods critically impact ACR performance across different models and paradigms.
Large Language Models Automated Code Review evaluation benchmarks cross-file context AI-assisted annotation expert-verified defect coverage LLM evaluation context granularity retrieval methods Agent architecture