PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Review

TL;DR

This work addresses the misalignment between static benchmarks and open-world deployment by introducing PeerRank, a fully autonomous, multi-agent framework in which LLMs generate evaluation tasks, answer with live web grounding, judge peers, and aggregate results into bias-aware rankings. The approach uses category-scoped web grounding to emulate real-world information gathering while keeping judging blind to retrieved evidence, and it explicitly quantifies self, name, and position biases across judges. In a large-scale study with 12 models and 420 endogenous questions, PeerRank yields stable, discriminative rankings that correlate with objective correctness on external benchmarks: TruthfulQA ($r=0.904$, $\rho=0.881$) and GSM8K ($r=0.873$, $\rho=0.763$). The results argue for bias-aware, open-world evaluation as a scalable complement to static benchmarks, while highlighting how judgment dynamics and deliberation effort shape model assessment.

Abstract

Evaluating large language models typically relies on human-authored benchmarks, reference answers, and human or single-model judgments, approaches that scale poorly, become quickly outdated, and mismatch open-world deployments that depend on web retrieval and synthesis. We introduce PeerRank, a fully autonomous end-to-end evaluation framework in which models generate evaluation tasks, answer them with category-scoped live web grounding, judge peer responses and aggregate dense peer assessments into relative performance estimates, without human supervision or gold references. PeerRank treats evaluation as a multi-agent process where each model participates symmetrically as task designer, respondent, and evaluator, while removing biased judgments. In a large-scale study over 12 commercially available models and 420 autonomously generated questions, PeerRank produces stable, discriminative rankings and reveals measurable identity and presentation biases. Rankings are robust, and mean peer scores agree with Elo. We further validate PeerRank on TruthfulQA and GSM8K, where peer scores correlate with objective accuracy. Together, these results suggest that bias-aware peer evaluation with selective web-grounded answering can scale open-world LLM assessment beyond static and human curated benchmarks.

Figures (21)

• Figure 1: Fully endogenous PeerRank evaluation pipeline (no human input). Models generate questions, answer with web grounding, evaluate peers under bias-control protocols, and aggregate scores into rankings and bias metrics. We additionally validate real-world truthfulness by running PeerRank on TruthfulQA lin2021truthfulqa and correlating peer scores with objective accuracy.
• Figure 2: PeerRank evaluation pipeline. Models act symmetrically as question generators, respondents, and judges. Answers are generated with web access enabled, while evaluation is performed without web access with bias quantification. Scores are aggregated into peer rankings, bias measurements, and judge statistics.
• Figure 3: Sample question and its evaluation
• Figure 4: Peer rankings (shuffle+blind): mean peer scores $\pm1$ SD; self-ratings excluded.
• Figure 5: Model performance by question category (shuffle+blind): mean peer score by category.