Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models

TL;DR

This work tackles the challenge of scalable, bias-resistant benchmarking for large language models by introducing Decentralized Arena (De-Arena), a fully automatic framework in which all participating LLMs act as judges. It combines a coarse-to-fine incremental ranking algorithm with an automatic representative-question selection strategy, enabling efficient, pairwise, democratic evaluation across many models and dimensions, with complexity framed as $\\mathcal{O}(k n \\log n)$. Across 66 LLMs and nine fine-grained dimensions, De-Arena achieves up to 0.97 correlation with human judgments (Chatbot Arena) while reducing annotation costs and mitigating single-model judge bias. The approach advances scalable, transparent LLM benchmarking and holds promise for broad real-world deployment and future automatic dimension discovery.

Abstract

The recent explosion of large language models (LLMs), each with its own general or specialized strengths, makes scalable, reliable benchmarking more urgent than ever. Standard practices nowadays face fundamental trade-offs: closed-ended question-based benchmarks (eg MMLU) struggle with saturation as newer models emerge, while crowd-sourced leaderboards (eg Chatbot Arena) rely on costly and slow human judges. Recently, automated methods (eg LLM-as-a-judge) shed light on the scalability, but risk bias by relying on one or a few "authority" models. To tackle these issues, we propose Decentralized Arena (dearena), a fully automated framework leveraging collective intelligence from all LLMs to evaluate each other. It mitigates single-model judge bias by democratic, pairwise evaluation, and remains efficient at scale through two key components: (1) a coarse-to-fine ranking algorithm for fast incremental insertion of new models with sub-quadratic complexity, and (2) an automatic question selection strategy for the construction of new evaluation dimensions. Across extensive experiments across 66 LLMs, dearena attains up to 97% correlation with human judgements, while significantly reducing the cost. Our code and data will be publicly released on https://github.com/maitrix-org/de-arena.

Figures (7)

• Figure 1: The overview of our method, consisting of the automatic representative question selection strategy (left) and the coarse-to-fine incremental ranking algorithm (right). Here, we show an example that creates a new dimension based on existing open-source datasets, and one of the insert iterations for adding the model Yi into the previous ranking list.
• Figure 2: (a) Spearman correlation between different LLM benchmarks in the overall dimension. (b)Benchmark cost and performance comparison in the overall dimension, where we show the average judge counts of each model and the correlation with Chatbot Arena.
• Figure 3: Spearman correlation with Chatbot Arena across varying judge model number.
• Figure 4: (a) De-Arena's mean (blue curve) and variance (shaded area) of Chatbot Arena correlation in the MT-bench dimension, with the increase in LLM number. (b) The distribution map of the LLM comparison counts in the MT-Bench dimension.
• Figure 5: (a) Binary search ranking differences between binary search and ground truth in four dimensions. (b) Spearman correlation with Chatbot Arena for different question selection methods.