What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review

TL;DR

The paper develops a process-centric analysis of modern peer review using OpenReview data from ICLR 2017–2025, integrating manuscript structure, submission/external signals, interaction logs, and sentiment and disagreement signals. It shows that the rebuttal stage, early submission, synchronized arXiv posting, and reproducibility resources robustly increase acceptance, often outweighing initial reviewer skepticism, while writing style and recent references modulate perceived novelty and contribution. Across dimensions, the study treats peer review as a signaling game where diverse signals update beliefs about quality, with distinct patterns across score regimes and an observed shift in style preferences in the post-LLM era. The findings yield practical guidelines for authors, reviewers, and meta-reviewers to enhance transparency, fairness, and efficiency, and they release a longitudinal dataset to support further research. Overall, the work demonstrates that process signals are essential for understanding and improving contemporary peer review.

Abstract

Peer review is the primary mechanism for evaluating scientific contributions, yet prior studies have mostly examined paper features or external metadata in isolation. The emergence of open platforms such as OpenReview has transformed peer review into a transparent and interactive process, recording not only scores and comments but also rebuttals, reviewer-author exchanges, reviewer disagreements, and meta-reviewer decisions. This provides unprecedented process-level data for understanding how modern peer review operates. In this paper, we present a large-scale empirical study of ICLR 2017-2025, encompassing over 28,000 submissions. Our analysis integrates four complementary dimensions, including the structure and language quality of papers (e.g., section patterns, figure/table ratios, clarity), submission strategies and external metadata (e.g., timing, arXiv posting, author count), the dynamics of author-reviewer interactions (e.g., rebuttal frequency, responsiveness), and the patterns of reviewer disagreement and meta-review mediation (e.g., score variance, confidence weighting). Our results show that factors beyond scientific novelty significantly shape acceptance outcomes. In particular, the rebuttal stage emerges as a decisive phase: timely, substantive, and interactive author-reviewer communication strongly increases the likelihood of acceptance, often outweighing initial reviewer skepticism. Alongside this, clearer writing, balanced visual presentation, earlier submission, and effective resolution of reviewer disagreement also correlate with higher acceptance probabilities. Based on these findings, we propose data-driven guidelines for authors, reviewers, and meta-reviewers to enhance transparency and fairness in peer review. Our study demonstrates that process-centric signals are essential for understanding and improving modern peer review.

Figures (11)

• Figure 1: Distribution of scores ($\bar{r}_i$) and score variance ($\sigma_i^2$)
• Figure 2: Variance and acceptance by score group and decision
• Figure 3: Distribution of reviewer confidence in borderline cases
• Figure 4: Acceptance rates by dominant sentiment in borderline
• Figure 5: Rebuttal dynamics. (a) Low/Borderline: faster and longer replies ↑ acceptance; High: no clear effect. (b) Depth ($Depth_i$) and count ($N_i$): helpful in Low, but neutral/negative when excessive in Borderline/High.