Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models

TL;DR

The paper empirically evaluates frontier open-weight LLMs as pre-review agents by comparing generated reviews to ground-truth OpenReview scores and linking them to post-publication metrics such as $Citations$, $Hit-papers$, $Novelty$, and $Disruption$. It introduces the $D_{LMRSD}$ dataset and analyzes two experiments (abstract-only and full-text reviews) across nine models, revealing strong overconfidence, inflation of scores, and weak alignment with human judgments, though some models show correlations with $C_5$ and hit-paper outcomes. Through ablations on perplexity, embedding similarity, and prompting strategies, the authors demonstrate limited gains in calibration and highlight substantial safety risks in autonomous deployment. The work argues for a cautious, tool-augmented, non-autonomous role for LLMs in peer review, and proposes an actionable framework of agentic review with multi-modal inputs and validated post-publication outcomes. By open-sourcing $D_{LMRSD}$ and outlining a rigorous evaluation pipeline, the study provides a concrete pathway to safer, more reliable use of LLMs in scientific evaluation.

Abstract

Large Language Models are versatile general-task solvers, and their capabilities can truly assist people with scholarly peer review as \textit{pre-review} agents, if not as fully autonomous \textit{peer-review} agents. While incredibly beneficial, automating academic peer-review, as a concept, raises concerns surrounding safety, research integrity, and the validity of the academic peer-review process. The majority of the studies performing a systematic evaluation of frontier LLMs generating reviews across science disciplines miss the mark on addressing the alignment/misalignment of reviews along with the utility of LLM generated reviews when compared against publication outcomes such as \textbf{Citations}, \textbf{Hit-papers}, \textbf{Novelty}, and \textbf{Disruption}. This paper presents an experimental study in which we gathered ground-truth reviewer ratings from OpenReview and used various frontier open-weight LLMs to generate reviews of papers to gauge the safety and reliability of incorporating LLMs into the scientific review pipeline. Our findings demonstrate the utility of frontier open-weight LLMs as pre-review screening agents despite highlighting fundamental misalignment risks when deployed as autonomous reviewers. Our results show that all models exhibit weak correlation with human peer reviewers (0.15), with systematic overestimation bias of 3-5 points and uniformly high confidence scores (8.0-9.0/10) despite prediction errors. However, we also observed that LLM reviews correlate more strongly with post-publication metrics than with human scores, suggesting potential utility as pre-review screening tools. Our findings highlight the potential and address the pitfalls of automating peer reviews with language models. We open-sourced our dataset $D_{LMRSD}$ to help the research community expand the safety framework of automating scientific reviews.

Figures (12)

• Figure 1: $D_{LMRSD}$ data distribution (rating score, and confidence of reviews)
• Figure 2: Token distribution across model families for the prompts evaluating scientific manuscripts
• Figure 3: Mean absolute error across different true score ranges for all models
• Figure 4: Confidence plot when LLMs review ideas in a given paper using abstract
• Figure 5: LLM-reported Confidence vs Prediction Bias when reviewing complete content presented in a scientific article using the full-text