Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

TL;DR

This work questions the narrative that peer review quality is in decline by developing a holistic, scalable framework to quantify review quality across venues and time. It introduces a unified representation of reviews (r^f and r^i) and pairs lightweight and LLM-based measurements across three quality criteria—Substantiveness, Actionability, and Grounding—to compute a composite quality score Q. The study finds no consistent decline in median review quality across ICLR, NeurIPS, and ARR campaigns, though it acknowledges variability and outlines five hypotheses and a roadmap for future empirical work. The framework enables cross-venue, longitudinal analyses and suggests practical directions for data collection, standardization, and methodological improvement in evaluating peer review quality at scale.

Abstract

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.

Figures (12)

• Figure 1: Framework overview. A review report $r$ is written for a paper $p$ by a reviewer $a$, and originates from a reviewing campaign $c$ that takes place as part of a venue $v$ at time $t$, e.g. ICLR-2018. Reviews differ in terms of their quality $Q(r)$, approximated through a range of measurements $m(r) \in M$ along a set of quality criteria. Aggregating review-level measurements allows comparative analysis of review quality across reviewing campaigns.
• Figure 2: Structural elements of a review report.
• Figure 3: Measurements examples. LEN, REQ and EXL are calculated on the flattened review (top). ITX is calculated on the Summary, Strengths and Weaknesses+ sections of the itemized review (bottom). ACT and GND averaged over the Weaknesses+ section.
• Figure 4: Metric correlations (Spearman’s $\rho$).
• Figure 5: $Q$ distribution on the study sample, all venues.