Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation

TL;DR

This work shows that LLMs used as evaluators can be systematically fooled by rewriting only an agent's chain-of-thought while keeping actions and observations fixed, causing substantial false positives in judging task success. Through a large, controlled study across 800 web-interaction trajectories and nine judges, the authors categorize CoT manipulation into style- and content-based strategies, with content-based Fabrication being the most effective. They demonstrate that prompt-based and rubric-based mitigations reduce but do not eliminate manipulation, and judge-time scaling offers only partial robustness at a significant compute cost, revealing a robustness-recall trade-off. The findings argue for evaluation mechanisms that verify reasoning against observable evidence and highlight the risk of reasoning-based reward hacking in current LLM-based evaluation pipelines. Overall, the paper advances a principled framework for studying CoT manipulation, provides a taxonomy of attack strategies, and underscores the need for grounding and verifiable evidence in agent evaluation.

Abstract

Large language models (LLMs) are increasingly used as judges to evaluate agent performance, particularly in non-verifiable settings where judgments rely on agent trajectories including chain-of-thought (CoT) reasoning. This paradigm implicitly assumes that the agent's CoT faithfully reflects both its internal reasoning and the underlying environment state. We show this assumption is brittle: LLM judges are highly susceptible to manipulation of agent reasoning traces. By systematically rewriting agent CoTs while holding actions and observations fixed, we demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks. We study manipulation strategies spanning style-based approaches that alter only the presentation of reasoning and content-based approaches that fabricate signals of task progress, and find that content-based manipulations are consistently more effective. We evaluate prompting-based techniques and scaling judge-time compute, which reduce but do not fully eliminate susceptibility to manipulation. Our findings reveal a fundamental vulnerability in LLM-based evaluation and highlight the need for judging mechanisms that verify reasoning claims against observable evidence.

Figures (24)

• Figure 1: CoT manipulation flips Gemini-2.5-Flash original judgment. The agent chooses the wrong check-in date but the manipulated CoT claims otherwise. The judge trusts the CoT over the observable evidence.
• Figure 2: An overview of our pipeline: We first collect trajectories by running a web agent on different tasks. Then, we rewrite the original CoTs from agent trajectories into unfaithful versions using different rewriting strategies without changing the actions or environment states.
• Figure 3: Browser Use success rates as measured by GLM-4.1V and GPT-4o judges. Despite achieving nearly identical true success, the agent using confident, manipulative reasoning receives substantially higher judged success rates, demonstrating that reasoning style alone can systematically bias LLM judges.
• Figure 4: Taxonomy of CoT manipulation strategies. We group CoT manipulations by the judgment heuristics they exploit: style-based strategies modify presentation without new claims, while content-based strategies inject misleading information. Actions and observations are held fixed throughout.
• Figure 5: Distribution of task categories across our evaluation suite. The 659 tasks span ten categories including booking, shopping, navigation, and information retrieval, with tasks drawn from existing benchmarks (WebArena, AssistantBench, WorkArena) and newly collected ones.