From Authors to Reviewers: Leveraging Rankings to Improve Peer Review
Weichen Wang, Chengchun Shi
TL;DR
This work tackles the challenge of noisy peer reviews amid massive ML conference submissions by proposing a reviewer-ranking–driven calibration framework. It models the score-generating process (SGP) where paper quality $\theta^*_i$ is perturbed by reviewer bias $b_r$ and variance $\sigma_r^2$, reviewers provide partial rankings via a Plackett–Luce model, and isotonic regression enforces consistency to recover $\theta^*_i$; a hierarchical ranking procedure yields a complete paper ordering from sparse comparisons. Through ICML-like simulations, the authors demonstrate that incorporating reviewer rankings significantly reduces RMSE compared to raw averages and can outperform author rankings, with a combined reviewer+author ranking approach offering the best accuracy in most scenarios. The method highlights fairness advantages by not depending on author submission counts and can be implemented by adjusting score processing rather than changing the review system, though it remains illustrative without formal theoretical guarantees. Overall, the study suggests practical pathways to enhance conference triage for large-scale submission environments such as NeurIPS.
Abstract
This paper is a discussion of the 2025 JASA discussion paper by Su et al. (2025). We would like to congratulate the authors on conducting a comprehensive and insightful empirical investigation of the 2023 ICML ranking data. The review quality of machine learning (ML) conferences has become a big concern in recent years, due to the rapidly growing number of submitted manuscripts. In this discussion, we propose an approach alternative to Su et al. (2025) that leverages ranking information from reviewers rather than authors. We simulate review data that closely mimics the 2023 ICML conference submissions. Our results show that (i) incorporating ranking information from reviewers can significantly improve the evaluation of each paper's quality, often outperforming the use of ranking information from authors alone; and (ii) combining ranking information from both reviewers and authors yields the most accurate evaluation of submitted papers in most scenarios.