League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models

TL;DR

This work introduces League of LLMs (LOL), a benchmark-free framework that mutual-evaluates multiple LLMs through dynamic, question-generation rounds and decentralized scoring to address data contamination, opacity, and subjective biases in traditional benchmarks. By replacing fixed datasets with self-generated tasks and leveraging Borda-based math scoring alongside absolute scores for programming, LOL achieves stable internal rankings (Top-$k$ consistency $= 70.7\%$) and reveals nuanced insights such as memorization-based answering and developer-family homophily biases. Across mathematics and programming, eight LLMs are compared, yielding discriminative capability rankings that correlate with established benchmarks ($ ho$ values ranging from ~0.71 to ~0.93). The approach demonstrates that mutual evaluation with reference answers can produce reliable, professional, and transparent assessments, and the authors provide public code to extend evaluation to new models and domains.

Abstract

Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical challenge due to data contamination, opaque operation, and subjective preferences. To address these issues, we propose League of LLMs (LOL), a novel benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. LOL integrates four core criteria (dynamic, transparent, objective, and professional) to mitigate key limitations of existing paradigms. Experiments on eight mainstream LLMs in mathematics and programming demonstrate that LOL can effectively distinguish LLM capabilities while maintaining high internal ranking stability (Top-$k$ consistency $= 70.7\%$). Beyond ranking, LOL reveals empirical findings that are difficult for traditional paradigms to capture. For instance, ``memorization-based answering'' behaviors are observed in some models, and a statistically significant homophily bias is found within the OpenAI family ($Δ= 9$, $p < 0.05$). Finally, we make our framework and code publicly available as a valuable complement to the current LLM evaluation ecosystem.

Figures (18)

• Figure 1: Mainstream LLM evaluation paradigms vs. League of LLMs (LOL). Compared under four core criteria: Dynamic, Transparent, Objective, and Professional.
• Figure 2: Overview of the League of LLMs evaluation pipeline. It consists of four phases: Generate Question, Answer Independently, Evaluate Mutually, and Update Ranking.
• Figure 3: Radar charts of LLM performance on mathematics tasks. They display the mean Borda scores of eight LLMs on eight questions, enabling comparison of their overall performance.
• Figure 4: Ability comparison of LLMs on programming tasks (question-solving vs. question-setting). They are measured by Own Average Score on Others' Questions and Others' Average Score on Own Question.
• Figure 6: Mathematics Scores. Rows are responders and columns are evaluators; each cell shows the mean score.