Survey on Evaluation of LLM-based Agents

TL;DR

The paper surveys evaluation methodologies for LLM-based agents, emphasizing the need for benchmarks that capture planning, tool use, self-reflection, and memory in dynamic environments. It catalogs application-specific benchmarks across web, software engineering, scientific, and conversational domains, alongside generalist-agent evaluations and evaluation frameworks. The analysis highlights trends toward realistic, live benchmarks and stresses gaps in safety, cost-efficiency, and fine-grained trajectory assessment, offering directions for future research. Overall, the work provides a comprehensive map of the current landscape and actionable guidance for developing robust, scalable, and responsible agent evaluations.

Abstract

The emergence of LLM-based agents represents a paradigm shift in AI, enabling autonomous systems to plan, reason, use tools, and maintain memory while interacting with dynamic environments. This paper provides the first comprehensive survey of evaluation methodologies for these increasingly capable agents. We systematically analyze evaluation benchmarks and frameworks across four critical dimensions: (1) fundamental agent capabilities, including planning, tool use, self-reflection, and memory; (2) application-specific benchmarks for web, software engineering, scientific, and conversational agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating agents. Our analysis reveals emerging trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks. We also identify critical gaps that future research must address-particularly in assessing cost-efficiency, safety, and robustness, and in developing fine-grained, and scalable evaluation methods. This survey maps the rapidly evolving landscape of agent evaluation, reveals the emerging trends in the field, identifies current limitations, and proposes directions for future research.

Figures (1)

• Figure 1: Overview of the paper.