LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluation

TL;DR

This work tackles the challenge of evaluating modern LLMs by replacing fixed-reference benchmarks with a decentralized mutual evaluation that leverages game-theoretic ranking. By having multiple LLMs act as both evaluators and evaluatees and aggregating their pairwise comparisons with algorithms like Kemeny-Young, the method aims to reproduce human judgments while reducing evaluator bias. Across diverse datasets and models, the approach generally improves alignment with human preferences and demonstrates bias mitigation, though performance varies by task type, being strongest in objective domains like math and code. The framework offers a scalable, human-aligned alternative to traditional benchmarks and highlights practical limitations and avenues for future exploration.

Abstract

Ideal or real - that is the question.In this work, we explore whether principles from game theory can be effectively applied to the evaluation of large language models (LLMs). This inquiry is motivated by the growing inadequacy of conventional evaluation practices, which often rely on fixed-format tasks with reference answers and struggle to capture the nuanced, subjective, and open-ended nature of modern LLM behavior. To address these challenges, we propose a novel alternative: automatic mutual evaluation, where LLMs assess each other's output through self-play and peer review. These peer assessments are then systematically compared with human voting behavior to evaluate their alignment with human judgment. Our framework incorporates game-theoretic voting algorithms to aggregate peer reviews, enabling a principled investigation into whether model-generated rankings reflect human preferences. Empirical results reveal both convergences and divergences between theoretical predictions and human evaluations, offering valuable insights into the promises and limitations of mutual evaluation. To the best of our knowledge, this is the first work to jointly integrate mutual evaluation, game-theoretic aggregation, and human-grounded validation for evaluating the capabilities of LLMs.

Figures (5)

• Figure 1: Illustration of game-theoretic peer evaluation for LLM performance ranking.
• Figure 2: The proposed framework for game-theoretic evaluation of LLMs.
• Figure 3: Alignment with Human Judgments under Different Evaluation Protocols. This figure reports the distribution of Pearson correlation coefficients between model-generated rankings and human preferences from Chatbot Arena under four evaluation protocols: SE, PE, SIE, and SFE. In each boxplot, a higher box indicates stronger alignment with human rankings, while a shorter box implies lower variance and thus more stable alignment. Additional results are provided in the appendix due to space constraints.
• Figure 4: Alignment with Human Judgments Across Benchmarks. Each boxplot shows the distribution of Pearson correlations between human rankings and rankings aggregated by the Kemeny-Young method under the micro-level setting. Higher values indicate stronger alignment. Purple dots represent macro-level correlations for each benchmark. “CreWrite” denotes the creative writing benchmark, and “InstFol” refers to instruction following.
• Figure 5: Alignment with Human Judgments under Different Evaluation Protocols. This figure reports the distribution of Pearson correlation coefficients between model-generated rankings and human preferences (from Chatbot Arena) across seven benchmarks under four evaluation protocols: SE (Self-Evaluation), PE (Peer Evaluation), SIE (Self-Inclusive Evaluation), and SFE (Self-Free Evaluation). In each boxplot, a higher box indicates stronger agreement with human rankings, while a shorter box implies lower variance and thus more stable alignment. Compared to SE and PE, the SIE and SFE protocols yield both higher and tighter boxes, suggesting that game-theoretic aggregation not only enhances ranking accuracy but also improves robustness against self-bias. Additional results are provided in the appendix due to space constraints.