Screening, sorting, and the feedback cycles that imperil peer review

TL;DR

The paper develops a low-dimensional, Adda–Ottaviani–inspired framework to explain a feedback loop that jeopardizes peer review: increasing submissions strain an unpaid reviewer pool, degrading review quality and sorting, which in turn prompts more submissions. It analyzes both a single elite journal and multiple competing elite journals, and extends the model with desk rejection to study how screening interventions alter welfare for authors, readers, and reviewers. Key findings show that reviewer capacity and review accuracy interact nonlinearly with submission volume, and that proliferation of journals can improve reader welfare at the cost of authors and reviewers, especially under noisy reviews. The work also discusses policy options—desk rejection optimization, cascaded or shared reviews, and perhaps compensating reviewers—that could slow or reverse the meltdown and improve overall scientific welfare.

Abstract

Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the long-term viability of peer review as an institution. What can or should be done to salvage it? Here, we develop mathematical models to reveal the intricate interactions among incentives faced by authors, reviewers, and readers in their endeavors to identify the best science. Two facets are particularly salient. First, peer review partially reveals authors' private sense of their work's quality through their decisions of where to send their manuscripts. Second, journals' reliance on traditionally unpaid and largely unrewarded review labor deprives them of a standard market mechanism -- wages -- to recruit additional reviewers when review labor is in short supply. We highlight a resulting feedback loop that threatens to overwhelm the peer review system: (1) an increase in submissions overtaxes the pool of suitable peer reviewers; (2) the accuracy of review drops because journals either must either solicit assistance from less qualified reviewers or ask current reviewers to do more; (3) as review accuracy drops, submissions further increase as more authors try their luck at venues that might otherwise be a stretch. We illustrate how this cycle is propelled by the increasing emphasis on high-impact publications, the proliferation of journals, and competition among these journals for peer reviews. Finally, we suggest interventions that could slow or even reverse this cycle of peer-review meltdown.

Figures (7)

• Figure 1: The peer-review meltdown cycle. Author screening and journal sorting interact in a feedback loop which inaccurate sorting loosens author screening adda2024grantmaking and looser screening makes sorting less accurate by depleting the pool of available review labor. Forces that exacerbate this feedback loop include (clockwise from top): Greater rewards to publishing lead more authors to submit their paper to top journals; a proliferation of journals gives authors more opportunities to obtain fresh reviews of already rejected manuscripts; journals' reliance on a shared pool of review labor compels journals to underuse desk rejection and overexploit the review pool; and noisier review reduces what authors can learn from having their papers rejected.
• Figure 2: Graphical illustration of the base model of Adda & Ottaviani adda2024grantmaking and associated welfare measures. A: Equilibrium conditions. The author-rationality condition dictates that the marginal author $\hat{q}$ has acceptance probability $c/v$, and the capacity-filling condition dictates that the area of the gold region---equal to the volume of accepted manuscripts---must equal the journal capacity $k$. Panel based on AO's Figs. 2 and 3 adda2024grantmaking. For this panel, $k=0.2$, $v/c = 5$, and $\sigma_X =\sigma_Y = 1$. B: Welfare measures for the same setting as panel (A). See text for details. C: Change in welfare under greater peer-review noise (in this case, $\sigma_Y = 2$; all other parameter settings as in panel B). D: Change in welfare as $v/c$ increases (in this case, $v/c = 10$; all other parameter settings as in panel B). E: $J=4$ elite journals. Jumps in the cost and benefit curves show the locations of marginal authors $\hat{q}_i$, $i=1,\ldots,4$. F: An infinity of elite microjournals and perfect author knowledge of their manuscript's quality.
• Figure 3: Author screening and journal sorting combine to determine how capably the journal identifies the most publication-worthy science. Points show a random sample of 200 manuscripts, located according to the author's sense of the paper's strength $X$ and by a reviewer's score $Y$. Point color indicates the quality of the manuscript $\theta$, with redder (bluer) values corresponding to higher- (lower-) quality manuscripts. The journal publishes $k=20\%$ of the papers, and filled symbols show the actual top 20% of manuscripts. At equilibrium, only authors who think their paper is at least as good as $\hat{x}$ (tan region) submit their paper, and the journal then accepts papers that reviewers rate as at least as good as $\hat{y}$ (intersection of tan and gray regions). The dashed horizontal line shows the acceptance threshold that the journal would use instead if every author submitted their paper. This figure uses $\sigma_x = 0.5$, $\sigma_Y = 1$, and $v/c = 5$.
• Figure 4: Feedback effects on peer-review load. The red, blue, and black curves show the review load $L$ induced by the review noise $\sigma_Y$ for three values of $v/c$. The dashed green curve shows how the review noise $\sigma_Y$ may respond to the review load $L$. Equilibria are found at the intersections of the red / blue / black curves with the green curve. A: A single elite journal and review noise $\sigma_Y$ that is independent of load $L$. B: A single elite journal and $\sigma_Y$ that increases with $L$. C: Three elite journals and $\sigma_Y$ that increases with $L$. Throughout, $\sigma_X = 1$ and $k=0.2$.
• Figure 5: Welfare consequences of journal proliferation. Increasing the number of elite journals helps readers but hurts authors and reviewers, and these effects are amplified when peer reviewing is less accurate. In this figure, the review noise, $\sigma_Y$, is independent of the review load, $L$, to highlight how the number of journals, $J$, interacts with review noise. Left: Review load per period vs. $v/c$ for $J=1$ (solid) or $J=3$ (dashed) journals, with review noise $\sigma_Y = 0.5$ (black) or $\sigma_Y = 1$ (red). Center: Author welfare. Right: Reader welfare, equal to the (standardized) average value of a published manuscript. Throughout, $k=0.2$ and $\sigma_X = 1$.

Theorems & Definitions (6)

• Claim
• proof
• Claim
• proof
• Claim
• proof