Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback

JungMin Yun, JuneHyoung Kwon, MiHyeon Kim, YoungBin Kim

TL;DR

The paper tackles the Reviewer Gap in AI conference peer review, characterized by a surge in submissions (Volume Gap) and a shortage of domain-expert reviewers (Quality Gap). It critiques approaches that rely on automated LLM-generated reviews and instead promotes LLMs as educational tools to augment human reviewers through a dual system: an LLM-assisted reviewer mentoring system and an LLM-assisted reviewer feedback system. Central to the approach is a unified five-principle rubric—Fidelity, Clarity, Fairness, Proportionality, and Constructiveness—that guides both training and feedback. The proposed framework emphasizes long-term reviewer development, structured learning stages, post-submission feedback with reliability checks, and human-in-the-loop governance, aiming to create a virtuous cycle of better reviews, improved research quality, and progressively smarter AI-assisted editorial processes.

Abstract

The rapid expansion of AI research has intensified the Reviewer Gap, threatening the peer-review sustainability and perpetuating a cycle of low-quality evaluations. This position paper critiques existing LLM approaches that automatically generate reviews and argues for a paradigm shift that positions LLMs as tools for assisting and educating human reviewers. We define the core principles of high-quality peer review and propose two complementary systems grounded in these foundations: (i) an LLM-assisted mentoring system that cultivates reviewers' long-term competencies, and (ii) an LLM-assisted feedback system that helps reviewers refine the quality of their reviews. This human-centered approach aims to strengthen reviewer expertise and contribute to building a more sustainable scholarly ecosystem.

Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback

TL;DR

The paper tackles the Reviewer Gap in AI conference peer review, characterized by a surge in submissions (Volume Gap) and a shortage of domain-expert reviewers (Quality Gap). It critiques approaches that rely on automated LLM-generated reviews and instead promotes LLMs as educational tools to augment human reviewers through a dual system: an LLM-assisted reviewer mentoring system and an LLM-assisted reviewer feedback system. Central to the approach is a unified five-principle rubric—Fidelity, Clarity, Fairness, Proportionality, and Constructiveness—that guides both training and feedback. The proposed framework emphasizes long-term reviewer development, structured learning stages, post-submission feedback with reliability checks, and human-in-the-loop governance, aiming to create a virtuous cycle of better reviews, improved research quality, and progressively smarter AI-assisted editorial processes.

Abstract

The rapid expansion of AI research has intensified the Reviewer Gap, threatening the peer-review sustainability and perpetuating a cycle of low-quality evaluations. This position paper critiques existing LLM approaches that automatically generate reviews and argues for a paradigm shift that positions LLMs as tools for assisting and educating human reviewers. We define the core principles of high-quality peer review and propose two complementary systems grounded in these foundations: (i) an LLM-assisted mentoring system that cultivates reviewers' long-term competencies, and (ii) an LLM-assisted feedback system that helps reviewers refine the quality of their reviews. This human-centered approach aims to strengthen reviewer expertise and contribute to building a more sustainable scholarly ecosystem.
Paper Structure (8 sections, 2 figures)

This paper contains 8 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Diagnosis of the Limitations in LLM-based Peer Review Systems
  3. Foundational Principles for Review Quality
  4. LLM-Assisted Reviewer Mentoring System
  5. LLM-Assisted Reviewer Feedback System
  6. Discussion & Alternative Views
  7. Conclusion
  8. Acknowledgments

Figures (2)

  • Figure 1: Overview of our proposed LLM-assisted reviewer mentoring and feedback system. The framework integrates foundational principles of high-quality reviews with a dual-system architecture comprising the mentoring system and the feedback system.
  • Figure 2: Macro-level virtuous cycle illustrating how improved reviews, refined research, and advanced AI technologies mutually reinforce the peer review ecosystem, supported by an internal collaborative human–AI improvement framework.