Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
Large language models (LLMs) are increasingly used as judges to evaluate agent performance, particularly in non-verifiable settings where judgments rely on agent trajectories including chain-of-thought (CoT) reasoning. This paradigm implicitly assumes that the agent's CoT faithfully reflects both its internal reasoning and the underlying environment state. We show this assumption is brittle: LLM judges are highly susceptible to manipulation of agent reasoning traces. By systematically rewriting agent CoTs while holding actions and observations fixed, we demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks. We study manipulation strategies spanning style-based approaches that alter only the presentation of reasoning and content-based approaches that fabricate signals of task progress, and find that content-based manipulations are consistently more effective. We evaluate prompting-based techniques and scaling judge-time compute, which reduce but do not fully eliminate susceptibility to manipulation. Our findings reveal a fundamental vulnerability in LLM-based evaluation and highlight the need for judging mechanisms that verify reasoning claims against observable evidence.
69822eda9084cb4f0ecb5486
Muhammad Khalifa
69822eda9084cb4f0ecb5487
Lajanugen Logeswaran
69822eda9084cb4f0ecb5488
Jaekyeom Kim
69822eda9084cb4f0ecb5489
Sungryull Sohn
69822eda9084cb4f0ecb548a
Yunxiang Zhang
69822eda9084cb4f0ecb548b
Moontae Lee
69822eda9084cb4f0ecb548c
Hao Peng
69822eda9084cb4f0ecb548d
Lu Wang
69822eda9084cb4f0ecb548e
Honglak Lee
Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
Large language models (LLMs) are increasingly used as judges to evaluate agent performance, particularly in non-verifiable settings where judgments rely on agent trajectories including chain-of-thought (CoT) reasoning. This paradigm implicitly assumes that the agent's CoT faithfully reflects both its internal reasoning and the underlying environment state. We show this assumption is brittle: LLM judges are highly susceptible to manipulation of agent reasoning traces. By systematically rewriting agent CoTs while holding actions and observations fixed, we demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks. We study manipulation strategies spanning style-based approaches that alter only the presentation of reasoning and content-based approaches that fabricate signals of task progress, and find that content-based manipulations are consistently more effective. We evaluate prompting-based techniques and scaling judge-time compute, which reduce but do not fully eliminate susceptibility to manipulation. Our findings reveal a fundamental vulnerability in LLM-based evaluation and highlight the need for judging mechanisms that verify reasoning claims against observable evidence.
Large language models used as judges for agent performance evaluation are vulnerable to manipulation of reasoning traces, with content-based fabrications being more effective than style-based alterations.
large language models agent performance chain-of-thought reasoning vision-language models manipulation strategies prompting-based techniques judge-time compute