Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FRONTIER-RevRec: A Large-scale Dataset for Reviewer Recommendation

Qiyao Peng, Chen Wang, Yinghui Wang, Hongtao Liu, Xuan Guo, Wenjun Wang

TL;DR

FRONTIER-RevRec addresses the need for large-scale, cross-disciplinary benchmarks in reviewer recommendation by constructing a dataset from authentic Frontiers peer-review records (2007–2025) that includes 177,941 reviewers and 478,379 papers across 209 journals and 1,736 sections. The authors systematically compare content-based, graph-based, and pure-text methods, finding that semantic, language-model–driven approaches (notably LLaMA2) outperform collaborative filtering, with aggregation strategies differing by pipeline stage (word-to-paper averaging vs. paper-to-reviewer LSTM). They also reveal fundamental structural differences between academic reviewer networks and commercial networks, such as fragmentation and long path lengths, which explain the limited efficacy of traditional collaborative filtering in this domain. The study provides a comprehensive benchmark and practical insights into effective reviewer recommendation, including the superiority of text-based signals and the nuanced role of aggregation methods, and outlines future directions for cross-domain and fairness-aware recommendations to improve scholarly peer review systems.

Abstract

Reviewer recommendation is a critical task for enhancing the efficiency of academic publishing workflows. However, research in this area has been persistently hindered by the lack of high-quality benchmark datasets, which are often limited in scale, disciplinary scope, and comparative analyses of different methodologies. To address this gap, we introduce FRONTIER-RevRec, a large-scale dataset constructed from authentic peer review records (2007-2025) from the Frontiers open-access publishing platform https://www.frontiersin.org/. The dataset contains 177941 distinct reviewers and 478379 papers across 209 journals spanning multiple disciplines including clinical medicine, biology, psychology, engineering, and social sciences. Our comprehensive evaluation on this dataset reveals that content-based methods significantly outperform collaborative filtering. This finding is explained by our structural analysis, which uncovers fundamental differences between academic recommendation and commercial domains. Notably, approaches leveraging language models are particularly effective at capturing the semantic alignment between a paper's content and a reviewer's expertise. Furthermore, our experiments identify optimal aggregation strategies to enhance the recommendation pipeline. FRONTIER-RevRec is intended to serve as a comprehensive benchmark to advance research in reviewer recommendation and facilitate the development of more effective academic peer review systems. The FRONTIER-RevRec dataset is available at: https://anonymous.4open.science/r/FRONTIER-RevRec-5D05.

FRONTIER-RevRec: A Large-scale Dataset for Reviewer Recommendation

TL;DR

FRONTIER-RevRec addresses the need for large-scale, cross-disciplinary benchmarks in reviewer recommendation by constructing a dataset from authentic Frontiers peer-review records (2007–2025) that includes 177,941 reviewers and 478,379 papers across 209 journals and 1,736 sections. The authors systematically compare content-based, graph-based, and pure-text methods, finding that semantic, language-model–driven approaches (notably LLaMA2) outperform collaborative filtering, with aggregation strategies differing by pipeline stage (word-to-paper averaging vs. paper-to-reviewer LSTM). They also reveal fundamental structural differences between academic reviewer networks and commercial networks, such as fragmentation and long path lengths, which explain the limited efficacy of traditional collaborative filtering in this domain. The study provides a comprehensive benchmark and practical insights into effective reviewer recommendation, including the superiority of text-based signals and the nuanced role of aggregation methods, and outlines future directions for cross-domain and fairness-aware recommendations to improve scholarly peer review systems.

Abstract

Reviewer recommendation is a critical task for enhancing the efficiency of academic publishing workflows. However, research in this area has been persistently hindered by the lack of high-quality benchmark datasets, which are often limited in scale, disciplinary scope, and comparative analyses of different methodologies. To address this gap, we introduce FRONTIER-RevRec, a large-scale dataset constructed from authentic peer review records (2007-2025) from the Frontiers open-access publishing platform https://www.frontiersin.org/. The dataset contains 177941 distinct reviewers and 478379 papers across 209 journals spanning multiple disciplines including clinical medicine, biology, psychology, engineering, and social sciences. Our comprehensive evaluation on this dataset reveals that content-based methods significantly outperform collaborative filtering. This finding is explained by our structural analysis, which uncovers fundamental differences between academic recommendation and commercial domains. Notably, approaches leveraging language models are particularly effective at capturing the semantic alignment between a paper's content and a reviewer's expertise. Furthermore, our experiments identify optimal aggregation strategies to enhance the recommendation pipeline. FRONTIER-RevRec is intended to serve as a comprehensive benchmark to advance research in reviewer recommendation and facilitate the development of more effective academic peer review systems. The FRONTIER-RevRec dataset is available at: https://anonymous.4open.science/r/FRONTIER-RevRec-5D05.
Paper Structure (25 sections, 7 figures, 8 tables)

This paper contains 25 sections, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reviewer Recommendation
  4. Existing Datasets
  5. FRONTIER-RevRec Dataset
  6. Dataset Construction
  7. Statistical Characteristic
  8. Data Distribution and Disciplinary Coverage
  9. Experiments
  10. Content-Based vs. Collaborative Signal Analysis
  11. Content-Based Analysis
  12. Graph-Based Analysis
  13. Compared with traditional recommendation datasets
  14. Benchmark Experiments on Reviewer Recommendation
  15. Experimental Setup
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Analysis of peer review process in FRONTIER-RevRec.
  • Figure 2: Distribution of Historical Reviewer Workload.
  • Figure 3: Comparison of clustering quality metrics across varying fuzziness parameters ($m$).
  • Figure 4: Paper-Reviewer Matching Framework Using Text Encoding.
  • Figure 5: Performance comparison of Dual-Tower and Single-Tower models.
  • ...and 2 more figures