Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System

TL;DR

The paper investigates how an Elo-ranked, multi-round review workflow with LLM agent reviewers can influence AC decisions and reviewer behavior. Using six personas and a memory-augmented framework, it simulates 30 rounds of ICLR-2025 submissions to study how Elo feedback and visibility alter strategy, accuracy, and bias. Key findings show that Elo improves AC decision accuracy and induces stratification toward high- and low-performing reviewers, while visible Elo can trigger strategic adaptation that may not improve substantive review quality; low-effort behaviors are penalized. The work highlights trade-offs between stability and diversity in peer review and demonstrates the value and challenges of agent-based simulations for designing future review systems, with practical implications for transparency and incentive design.

Abstract

In this work, we explore the Large Language Model (LLM) agent reviewer dynamics in an Elo-ranked review system using real-world conference paper submissions. Multiple LLM agent reviewers with different personas are engage in multi round review interactions moderated by an Area Chair. We compare a baseline setting with conditions that incorporate Elo ratings and reviewer memory. Our simulation results showcase several interesting findings, including how incorporating Elo improves Area Chair decision accuracy, as well as reviewers' adaptive review strategy that exploits our Elo system without improving review effort. Our code is available at https://github.com/hsiangwei0903/EloReview.

Figures (3)

• Figure 1: Our work explores the LLM agent dynamic of Elo-ranked Review System where reviewer is able to adjust their review strategy across review rounds.
• Figure 2: Four stages of our proposed Elo-ranked paper review process.
• Figure 3: Elo rating dynamics of different reviewer personas across three experiment setups.