Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review

TL;DR

This paper tackles the reliability and scalability of open, bottom-up peer review by analyzing data from CCN2023 and ICLR2023, where high variability and low inter-reviewer agreement challenge traditional consensus. It develops a Bayesian framework that weights individual reviews by empirically estimated reviewer quality, showing significant gains over simple averaging and revealing that reviewer quality is independent of authorship quality, with intermediate-author reviewers performing best. It further proposes an open post-publication platform where users rate both papers and reviewers, supported by a generative model and a Bayes-weighted scoring approach; this framework remains robust to unreliable reviewers and bots when proper uncertainty thresholds and verification are in place, and direct ratings of reviews can improve accuracy. To incentivize participation and broad coverage, the authors discuss extrinsic mechanisms such as reviewer-quality publicization, scarcity-based caps, and information-gain based rewards, arguing that a well-designed mix can yield scalable, fair, and transparent peer assessment with faster feedback loops.

Abstract

Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and undermine public trust. Here, we propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process. First, using data from two major scientific conferences (CCN2023 and ICLR2023), we highlight how high variability of review scores and low correlation across reviewers presents a challenge for collective review. We quantify reviewer agreement with community consensus scores and use this as a reviewer quality estimator, showing that surprisingly, reviewer quality scores are not correlated with authorship quality. Instead, we reveal an inverted U-shape relationship, where authors with intermediate paper scores are the best reviewers. We assess empirical Bayesian methods to estimate paper quality based on different assessments of individual reviewer reliability. We show how under a one-shot review-then-score scenario, both in our models and on real peer review data, a Bayesian measure significantly improves paper quality assessments relative to simple averaging. We then consider an ongoing model of publishing, reviewing, and scoring, with reviewers scoring not only papers but also other reviewers. We show that user-generated reviewer ratings can yield robust and high-quality paper scoring even when unreliable (but unbiased) reviewers dominate. Finally, we outline incentive structures to recognize high-quality reviewers and encourage broader reviewing coverage of submitted papers. These findings suggest that a self-selecting open peer review process is potentially scalable, reliable, and equitable with the possibility of enhancing the speed, fairness, and transparency of the peer review process.

Figures (11)

• Figure 1: High variability in paper scores and low correlation across reviewers are challenges for collective paper review. (A) Correlation of scores between pairs of reviewers reviewing the same submission in the conference Cognitive Computational Neuroscience 2023 ("Impact" scores). (B) Common score-normalizing techniques -- z-scoring, ranking, removing the mean, and inverting the distribution, applied to the CCN2023 data: the correlations remain low. (C) Corresponding to (A) analysis for the International Conference on Learning Representations 2023. (D) In ICLR2023 data, pairs of ratings with higher self-reported confidence scores have a higher correlation, but only a minority of reviews is tagged as high-confidence. For all panels showing correlation, $p<0.001$.
• Figure 2: In a simple model, weighting reviews using an empirical reviewer quality metric results in better estimation of paper quality. (A) The simple model: different reviewers all assign scores to the same paper according to a distribution centered at the ground truth quality (hidden), but with different standard deviations, corresponding to different reviewer accuracies. (B) Bayesian weighting of review scores based on reviewer quality leads to a tighter estimation of the ground truth paper quality scores. Curves shown for ground truth reviewer quality (black solid line), empirically estimated reviewer quality based on five previous reviews (black dashed line), and simple mean score (solid gray line). Shaded regions denote the s.d.
• Figure 3: Reviewer quality, unrelated to author quality, improved paper quality estimation in CCN2023 submissions. (A) In the CCN2023 review process, reviewer quality (measured as MSD of user's review scores from community average score across all reviewed papers) and author quality (measured as the average score the user got for their paper) were unrelated. To preserve reviewer privacy in this visualization, some outlier points were removed from the plot and reviewers were binned together in small bins. (B) Using estimated author quality as a stand-in for reviewer quality as the Bayes weight did not improve the estimate of paper quality relative to baseline (cross-validated results, shown on withheld papers). Error bars denote the s.e.m. (C) Users with higher estimated reviewer quality (Bayes weight estimated on the training data) consistently had smaller deviations from community score on withheld papers when estimating reviewer quality. Shaded regions denote the s.e.m. (D) Using the estimated reviewer quality as Bayes weights led to consistently smaller deviations from the community score on withheld papers. Error bars denote the s.e.m.
• Figure 4: Reviewing reviews: Benefits of reviewer quality metrics in a richer ecosystem. (A) An open platform where content is published immediately, reviewers are self-selected, and the community assesses (rates) review quality. (B) Estimation of reviewer accuracy: binning reviewers by the ratings their reviews received enables estimation of that group's accuracy in paper evaluation (mitigating the problem of estimating an accuracy score based on individual reviewer scores, given the small number of reviews supplied by each reviewer). (C) Bayes weighting of review scores based on reviewer accuracy as estimated from (B) leads to a better estimate of the ground truth paper quality scores (left; solid black line) than simple averaging of all review scores (left; gray line). An alternative heuristic of simply thresholding out all reviewers with a sufficiently low rating percentile (here, below 80 percentile, red line) results in a similar correlation to Bayes weighting but much lower coverage of papers on the platform (right). (D) Distributions of estimated reviewer quality scores for bots and real users in the simulated platform. In high-noise regime with most of the users being bots (up to 80%), reliability of the reviewer quality estimation breaks down, suggesting that alternative ways of excluding bots might be more effective.
• Figure 5: Binarized scoring of reviews does not reduce performance.Left: The baseline condition, where all scores and reviews are real numbers between 0 and 1. Center: The "binary scores" condition, where the reviews of reviews (but not paper review scores themselves) are binarized using a threshold of 0.5, does not strongly reduce performance. Right: The binary scores condition, where the number of binary review of reviews scores are 5 times as numerous as on the left (under the assumption that it's significantly easier to provide a binary label than a real score), leads to improved performance relative to fewer non-binary scores, ***p$<$0.0001.