You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism
Weijie Su
TL;DR
The Isotonic Mechanism introduces owner-assisted elicitation by requiring authors to rank their submissions, then calibrates raw review scores via isotonic regression to yield $\widehat{\bm{R}}^{\pi}$. It proves that truthful reporting of the ground-truth ranking is optimal under a convex, nondecreasing utility, and shows that the adjusted scores consistently improve ground-truth estimation, with pronounced gains when $n$ and noise are large. The framework also characterizes when truthful elicitation is possible and proves the isotonic partition is the finest truthful partition among partition-based mechanisms, while offering several extensions (local and coarse rankings, robustness, multi-author settings). Practical experiments at ICML and the possibility of broader applications motivate the approach, alongside theoretical links to majorization, Schur-convexity, and minimax estimation. Overall, the paper provides a rigorous, statistically grounded mechanism to enhance peer review by incorporating authors' private assessments into the scoring process. The resulting methodology yields provable truthfulness, improved estimation accuracy, and clear pathways for implementation and further theory development.
Abstract
Machine learning (ML) and artificial intelligence (AI) conferences including NeurIPS and ICML have experienced a significant decline in peer review quality in recent years. To address this growing challenge, we introduce the Isotonic Mechanism, a computationally efficient approach to enhancing the accuracy of noisy review scores by incorporating authors' private assessments of their submissions. Under this mechanism, authors with multiple submissions are required to rank their papers in descending order of perceived quality. Subsequently, the raw review scores are calibrated based on this ranking to produce adjusted scores. We prove that authors are incentivized to truthfully report their rankings because doing so maximizes their expected utility, modeled as an additive convex function over the adjusted scores. Moreover, the adjusted scores are shown to be more accurate than the raw scores, with improvements being particularly significant when the noise level is high and the author has many submissions -- a scenario increasingly prevalent at large-scale ML/AI conferences. We further investigate whether submission quality information beyond a simple ranking can be truthfully elicited from authors. We establish that a necessary condition for truthful elicitation is that the mechanism be based on pairwise comparisons of the author's submissions. This result underscores the optimality of the Isotonic Mechanism, as it elicits the most fine-grained truthful information among all mechanisms we consider. We then present several extensions, including a demonstration that the mechanism maintains truthfulness even when authors have only partial rather than complete information about their submission quality. Finally, we discuss future research directions, focusing on the practical implementation of the mechanism and the further development of a theoretical framework inspired by our mechanism.