Agent-as-a-Judge
LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge, where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation.
69606e265b7998385e639497
Runyang You
claimed_verified
2026-01-09T08:35:17.609Z
64e14c5b12a5504dda70e60d
/avatars/944b486bb037364ef7d9d2c826526708.svg
Runyang
dd101bb
user
dd101bb
69606e265b7998385e639498
Hongru Cai
claimed_verified
2026-01-09T15:46:07.920Z
653a111eee5888edef9182cf
https://cdn-avatars.huggingface.co/v1/production/uploads/653a111eee5888edef9182cf/7jQ08JDk2UEBla91At8QR.jpeg
Hongru Cai
HongruCai
user
HongruCai
69606e265b7998385e639499
Caiqi Zhang
69606e265b7998385e63949a
Qiancheng Xu
69606e265b7998385e63949b
Meng Liu
69606e265b7998385e63949c
Tiezheng Yu
69606e265b7998385e63949d
Yongqi Li
69606e265b7998385e63949e
Wenjie Li
Agent-as-a-Judge
LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge, where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation.
Large language models face limitations in evaluating complex, multi-step tasks, prompting the development of agent-based evaluation systems that utilize planning, tool-augmented verification, and multi-agent collaboration for more robust assessments.
LLM-as-a-Judge Agent-as-a-Judge agentic judges planning tool-augmented verification multi-agent collaboration persistent memory evaluation systems agentic evaluation