Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

Weicong Liu, Zixuan Yang, Yibo Zhao, Xiang Li

TL;DR

This work tackles reviewer assignment under rapid topic shifts by providing LR-Bench, a high-fidelity benchmark built from 2024–2025 AI/NLP manuscripts with 1,055 expert-annotated familiarity ratings, and RATE, a reviewer-centric ranking framework that distills each reviewer’s publication history into keyword-based profiles and trains with annotation-free, dual-view weak supervision derived from heuristic retrieval signals. RATE uses LLMs to create unified reviewer profiles and employs BM25-based triplets for robust, annotation-free learning, achieving state-of-the-art performance on LR-Bench and the CMU gold-standard dataset with strong generalization across backbones. Empirical results show RATE surpasses traditional heuristics and many embedding/LLM baselines, and human evaluations corroborate practical utility in selecting high-quality reviewer sets. The work advances scalable, up-to-date evaluation and matching in peer review systems, while acknowledging limitations such as potential noise in LLM-derived profiles and the need to consider author-order signals for certain domains.

Abstract

Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling matching each manuscript against a reviewer profile directly. Across LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/datasets/Gnociew/LR-bench, and a GitHub repository at https://github.com/Gnociew/RATE-Reviewer-Assign.

RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

TL;DR

This work tackles reviewer assignment under rapid topic shifts by providing LR-Bench, a high-fidelity benchmark built from 2024–2025 AI/NLP manuscripts with 1,055 expert-annotated familiarity ratings, and RATE, a reviewer-centric ranking framework that distills each reviewer’s publication history into keyword-based profiles and trains with annotation-free, dual-view weak supervision derived from heuristic retrieval signals. RATE uses LLMs to create unified reviewer profiles and employs BM25-based triplets for robust, annotation-free learning, achieving state-of-the-art performance on LR-Bench and the CMU gold-standard dataset with strong generalization across backbones. Empirical results show RATE surpasses traditional heuristics and many embedding/LLM baselines, and human evaluations corroborate practical utility in selecting high-quality reviewer sets. The work advances scalable, up-to-date evaluation and matching in peer review systems, while acknowledging limitations such as potential noise in LLM-derived profiles and the need to consider author-order signals for certain domains.

Abstract

Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling matching each manuscript against a reviewer profile directly. Across LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/datasets/Gnociew/LR-bench, and a GitHub repository at https://github.com/Gnociew/RATE-Reviewer-Assign.
Paper Structure (37 sections, 3 equations, 5 figures, 12 tables)

This paper contains 37 sections, 3 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reviewer Assignment Benchmarks
  4. Reviewer Assignment Methods
  5. Dataset Construction
  6. Data Source & Preprocessing
  7. Query Sampling & Candidate Recall
  8. Expertise Ground Truth Collection
  9. Benchmark Construction & Statistics
  10. Reviewer-centric Ranking
  11. Reviewer Profiling
  12. Dual-view Heuristic Preference Alignment
  13. Experiments
  14. Experimental Setup
  15. Dataset.
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Challenges of reviewer assignment.
  • Figure 2: Overview of pipline.
  • Figure 10: Prompting template for LLM-based author disambiguation.
  • Figure 11: Prompting template for LLM-based academic keyword extraction.
  • Figure 12: Zero-shot prompting template for LLM-based evaluation.