Eliciting Honest Information From Authors Using Sequential Review

TL;DR

The paper tackles the challenge of unreliable conference peer review by introducing a sequential review mechanism that truthfully elicits authors’ rankings of their papers under a setting where an author’s utility is non-decreasing in the true quality of accepted papers. It develops a general framework with monotone acceptance, review, and transition mappings, proving truthfulness under these conditions, and offers two practical instantiations: the memoryless coin-flip and the credit pool mechanisms. Through both Gaussian-noise simulations and real-I CLR OpenReview data, the authors show substantial gains in conference utility and reductions in review burden compared to a baseline parallel review, while also demonstrating incentives for authors to focus on high-quality output. They further analyze endogenous paper quality, showing that the sequential approach encourages higher-quality, fewer papers, and discuss limitations like coauthorship and delay trade-offs as well as broader applications to other principal-agent settings. Overall, the work provides a principled mechanism design framework with practical implications for improving both the efficiency and quality of peer review in large conferences.

Abstract

In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in peer review settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) improving the quality of accepted papers, reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.

Figures (6)

• Figure 1: The sequential review mechanism in round $i$.
• Figure 2: The relative conference utility under different parameter settings. Unless otherwise specified, the default parameter setting is $\varphi^G= (n=5, \mu_q = -1, \sigma_q=2, \sigma_r = 1)$.
• Figure 3: The relative conference utility and the relative review burden under different parameter settings. Unless otherwise specified, the default parameter setting is $\varphi^G= (n=5, \mu_q = -1, \sigma_q=2, \sigma_r = 1)$.
• Figure 4: Empirical distributions of the number of papers each author has for ICLR 2021-2023. Papers with multiple authors are attributed to the author with the largest number of papers.
• Figure 5: Relative review burden of the threshold sequential review mechanism for ICLR 2021-2023.

Theorems & Definitions (27)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Theorem 4.1
• proof
• Proposition 4.2
• Proposition 4.3