Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Peer-Reviewing Worth the Effort?

Kenneth Church, Raman Chandrasekar, John E. Ortega, Ibrahim Said Ahmad

TL;DR

The paper reframes peer-review as a forecasting problem for future citations and compares reliance on venue against early citation signals. It shows that early citations $\rho$ are more predictive of future impact than venue across ACL, PubMed, and ArXiv, and demonstrates this with correlations and a regression framework. The results indicate that conditioning on early returns yields higher $h$-indices and mean impacts than relying on venue alone, with strong robustness across data sources. To address scalability, the authors propose a DontDoIt (DDI) alternative and a nomination-based approach that leverages early signals to reduce reviewer burden while preserving quality and transparency.

Abstract

How effective is peer-reviewing in identifying important papers? We treat this question as a forecasting task. Can we predict which papers will be highly cited in the future based on venue and "early returns" (citations soon after publication)? We show early returns are more predictive than venue. Finally, we end with constructive suggestions to address scaling challenges: (a) too many submissions and (b) too few qualified reviewers.

Is Peer-Reviewing Worth the Effort?

TL;DR

The paper reframes peer-review as a forecasting problem for future citations and compares reliance on venue against early citation signals. It shows that early citations are more predictive of future impact than venue across ACL, PubMed, and ArXiv, and demonstrates this with correlations and a regression framework. The results indicate that conditioning on early returns yields higher -indices and mean impacts than relying on venue alone, with strong robustness across data sources. To address scalability, the authors propose a DontDoIt (DDI) alternative and a nomination-based approach that leverages early signals to reduce reviewer burden while preserving quality and transparency.

Abstract

How effective is peer-reviewing in identifying important papers? We treat this question as a forecasting task. Can we predict which papers will be highly cited in the future based on venue and "early returns" (citations soon after publication)? We show early returns are more predictive than venue. Finally, we end with constructive suggestions to address scaling challenges: (a) too many submissions and (b) too few qualified reviewers.

Paper Structure

This paper contains 23 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Prioritization as a Forecasting Task
  3. H-Index and Impact
  4. Related Work
  5. Metrics: H-index and Impact Factor
  6. Numerous Challenges to Reviewing
  7. Purpose of peer-reviewing?
  8. Validity and Reliability
  9. Vulnerabilities, Cheating and Ethics
  10. Related Work on Predicting Citations
  11. Improving Predictions
  12. Advice to Authors
  13. Predictions Based on Citations
  14. Correlations
  15. H-index and Impact
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Early Returns (left) $\gg$ Venue (right), based on correlations ($\rho$) from Tables \ref{['tab:cor_of_citations']}-\ref{['tab:cor_of_venues']}. Data is based on Semantic Scholar (S2) Wade2022TheSS, where the venue field refers not only to conferences, but also to journals and more.
  • Figure 2: Impact factor ($\mu$) from \ref{['tab:pubmed_stats']}. Simple rule of thumb: for most venues, reviewers are no better than 1+ early citations in terms of $\mu$; for all venues, reviewers are no better than 20+ early citations.
  • Figure 3: Boxplots of predictions from regression model for ACL papers. The bars are so narrow that they are hard to see on the left because early returns are more predictive than venue.