Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

TL;DR

This work tackles the challenge of selecting best-paper awards amid growing conference submissions by introducing an author-assisted mechanism that uses authors’ self-ranked submissions to adjust reviewer scores via isotonic regression. It proves truthfulness under relaxed utility conditions, notably allowing quota $k=1$ with nondecreasing utility, and empirically validates convexity of the best-paper utility using ICLR and NeurIPS data. The approach is extended to overlapping authors and tested in simulations showing substantial improvements in identifying high-quality papers, with Blind protocols offering robust practical performance. Overall, the method provides a transparent, resource-free augmentation to peer-review that can improve best-paper selections without overburdening reviewers. The results suggest broad applicability beyond conference settings, including grants and hiring where insiders bear reliable ordinal information about their own items.

Abstract

Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.

Figures (17)

• Figure 1: Comparison and second derivative of the regression of best paper probabilities across ICLR 2019---2023 and NeurIPS 2021---2023. See Section \ref{['sec:expe']} and Section \ref{['exp:NeurIPS']} for detailed analysis and additional results. Error bars show the binomial standard error of the mean (SEM), $\sqrt{p(1-p)/n}$, computed with bucket size $n$ (papers per score bin), where $p$ is the empirical best paper probability within each bin.
• Figure 2: Comparison of acceptance probabilities for ICLR 2021 and NeurIPS 2021.
• Figure 3: Comparison of Acceptance Probabilities at ICLR 2021 and ICLR 2022.
• Figure 4: Oral presentation probability at ICLR 2022. The saturation point is visibly shifted to the right, occurring at much higher scores compared to the general acceptance curve. This illustrates how a more stringent selection criterion alters the probability distribution.
• Figure 5: Acceptance probability at NeurIPS 2021. Similar to the ICLR data, this curve exhibits a clear saturation effect for high scores, demonstrating non-convexity in the standard acceptance regime.

Theorems & Definitions (14)

• Definition 1: marshallInequalitiesTheoryMajorization2011a
• Definition 2: suTruthfulOwnerAssistedScoring2022a
• Lemma 3: suTruthfulOwnerAssistedScoring2022a
• Definition 4: marshallInequalitiesTheoryMajorization2011a
• Lemma 5: dykstraMajorizationLorenzOrder1988a
• Definition 6
• Lemma 7: muirheadMethodsApplicableIdentities1902a
• Lemma 8: boydConvexOptimization2004a
• Lemma 9: (Schur-Convexity Criterion for Truthfulness)
• proof