Agent-as-a-Judge

TL;DR

This survey articulates the shift from LLM-as-a-Judge to Agent-as-a-Judge, arguing that planning, tool-augmented verification, memory, and multi-agent collaboration yield more robust, verifiable evaluations for complex, multi-step evaluands. It offers a taxonomy spanning five methodologies and outlines three developmental stages—Procedural, Reactive, and Self-Evolving—to organize this evolution. The review maps the methodologies to applications across general and professional domains, and candidly discusses challenges such as computational cost, latency, safety, and privacy, while presenting directions like personalization, generalization, interactivity, and optimization to guide future work. Collectively, the paper provides a concrete roadmap for advancing autonomous, verifiable agentic evaluators capable of adapting to the expanding capabilities and domains of AI systems.

Abstract

LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge, where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation.

Figures (4)

• Figure 1: Comparison between LLM-as-a-Judge (\ref{['subfig:llm-as-a-judge']}) and Agent-as-a-Judge (\ref{['subfig:agent-as-a-judge']}). The former performs direct single-pass evaluation, while the latter leverages planning, memory, and tool-augmented capabilities for enhanced evaluation.
• Figure 2: A taxonomy of Agent-as-a-Judge organizing Methodologies (§ \ref{['sec:methods']}) and Applications (§ \ref{['sec:apps']}). Background gradients illustrate the coverage of developmental stages, from Procedural to Reactive and then to Self-Evolving.
• Figure 3: Multi-agent collaboration paradigms.
• Figure 4: An overview of Agent-as-a-Judge application domains and their fine-grained task categories.