Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review

TL;DR

The paper quantitatively analyzes interaction effects between LLM use in papers and in reviews across major ML conferences using observational OpenReview data (≈125k paper–review pairs) and synthetic experiments. It employs a causal regression framework to quantify how LLM-aided reviews affect ratings of LLM-aided versus human-written papers, revealing that apparent leniency toward LLM-aided content is largely driven by paper quality and sample composition, while fully LLM-generated reviews exhibit severe rating compression. Metareviews with LLM assistance are associated with higher acceptance likelihood, though fully LLM-generated metareviews are harsher and less aligned with actual decisions, indicating the human in the loop moderates AI biases. Together, the findings offer nuanced guidance for policy design around LLM use in peer review and demonstrate that AI tools interact in complex ways with existing decision processes. The work highlights the need for careful governance to preserve discriminative power in peer review while leveraging AI-assisted workflows.

Abstract

There are increasing indications that LLMs are not only used for producing scientific papers, but also as part of the peer review process. In this work, we provide the first comprehensive analysis of LLM use across the peer review pipeline, with particular attention to interaction effects: not just whether LLM-assisted papers or LLM-assisted reviews are different in isolation, but whether LLM-assisted reviews evaluate LLM-assisted papers differently. In particular, we analyze over 125,000 paper-review pairs from ICLR, NeurIPS, and ICML. We initially observe what appears to be a systematic interaction effect: LLM-assisted reviews seem especially kind to LLM-assisted papers compared to papers with minimal LLM use. However, controlling for paper quality reveals a different story: LLM-assisted reviews are simply more lenient toward lower quality papers in general, and the over-representation of LLM-assisted papers among weaker submissions creates a spurious interaction effect rather than genuine preferential treatment of LLM-generated content. By augmenting our observational findings with reviews that are fully LLM-generated, we find that fully LLM-generated reviews exhibit severe rating compression that fails to discriminate paper quality, while human reviewers using LLMs substantially reduce this leniency. Finally, examining metareviews, we find that LLM-assisted metareviews are more likely to render accept decisions than human metareviews given equivalent reviewer scores, though fully LLM-generated metareviews tend to be harsher. This suggests that meta-reviewers do not merely outsource the decision-making to the LLM. These findings provide important input for developing policies that govern the use of LLMs during peer review, and they more generally indicate how LLMs interact with existing decision-making processes.

Figures (14)

• Figure 1: (a) LLM use trends over time at ICLR. (b-c) Rating distributions for human vs. LLM-aided papers and reviews for ICLR 2024 and 2025 submissions. LLM-aided papers receive a higher distribution of lower scores while LLM-aided reviews more frequently give middling scores relative to their fully human counterparts. Here LLM-aided means the text showed sufficient similarity to LLM vocabulary ($\alpha>0.15$ in Section \ref{['sec:llmdetectionmethod']}).
• Figure 2: Causal diagram for LLM use in peer review. $T_{ij}$: treatment (LLM-assisted review). $Y_{ij}$: outcome (rating). $X_i^{\text{LLM}}$: paper LLM use. $X_i^{\text{area}}$: paper area (confounder). Solid arrows are hypothesized causal effects; dashed arrows are potential unobserved confounding.
• Figure 3: Average ratings from LLM-aided reviews vs. human reviews across quality buckets. Shaded area is 1.96*std-error.
• Figure 4: Regression coefficients by quality bucket. Each point represents a separate regression run on papers within that quality bucket. Shaded area is 95% CI. The dashed red line shows the estimate from the full regression for comparison. Zero is indicated with a black dotted line.
• Figure 5: Average Rating provided by Human reviews and fully LLM reviews to randomly sampled ICLR 2025 papers, bucketed by quality. The datapoints correspond to quality bucket 10 are noisy because only 1 such paper was sampled.