CATArena: Evaluation of LLM Agents through Iterative Tournament Competitions

TL;DR

This work addresses the need for evaluating LLM-driven agents beyond fixed end-to-end tasks by introducing an iterative peer-learning framework and CATArena, a tournament-style benchmark with four open-ended board and card games. The authors define rigorous metrics—Strategy Coding, Global Learning, Counter-Adaptation, Self-Improvement, and Generalizability—computed from a dynamic scoring matrix to measure both coding and learning capabilities. Through experiments on minimal and commercial code agents, they demonstrate reliable, scalable evaluation and reveal distinct learning dynamics, including when and how agents generalize to variant rules. The framework's open-ended nature and extensibility hold promise for continual benchmarking as agent capabilities evolve, enabling deeper insight into the mechanisms of learning and strategy formation in LLM-powered agents.

Abstract

Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools. However, current benchmarks mainly assess end-to-end performance in fixed scenarios, restricting evaluation to specific skills and suffering from score saturation and growing dependence on expert annotation as agent capabilities improve. In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence. We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback, thereby systematically evaluating their learning capabilities. To address the score saturation issue in current benchmarks, we introduce CATArena, a tournament-style evaluation platform featuring four diverse board and card games with open-ended scoring. By providing tasks without explicit upper score limits, CATArena enables continuous and dynamic evaluation of rapidly advancing agent capabilities. Experimental results and analyses involving both minimal and commercial code agents demonstrate that CATArena provides reliable, stable, and scalable benchmarking for core agent abilities, particularly learning ability and strategy coding.

Figures (10)

• Figure 1: Overview of the evaluation framework and CATArena. The evaluation framework adopts an iterative peer-learning based competitive process. In round 1, LLM agents develop initial strategies via coding. These strategies are matched in a tournament arena, producing rankings and logs. In each subsequent round, agents analyze previous codes and logs, refine their strategies, and compete again. CATArena includes four open-ended and rankable games to cover diverse settings. Based on the tournament results from all rounds, a carefully designed scoring matrix and evaluation scheme are used to robustly quantify various agent abilities.
• Figure 2: Visualization of agents' learning patterns and scores.. For clarity, we use family names to represent LLM models instead of their full names. The results for other games are in Appendix \ref{['app:learning']}.
• Figure 3: Action consistency between agents' code and LLM-Players on Chess endgames.
• Figure 4: Trend of global performance score $G_i^n$ in Gomoku, Hold'em, Bridge and Chess.
• Figure 5: Action consistency between round 1 and round 2 agents’ code on endgames.