EchoReview: Learning Peer Review from the Echoes of Scientific Citations

TL;DR

The paper tackles the scalability and subjectivity of traditional peer review by proposing EchoReview, a citation-context-driven data synthesis framework that converts long-term community feedback into structured review-style data. It builds EchoReview-16K, a cross-conference, cross-year dataset, and EchoReview-Bench to evaluate automated reviewers, culminating in EchoReviewer-7B trained with LoRA. Empirical results show improvements in evidence support and review comprehensiveness, with analyses of time-span effects and cross-paradigm synergy indicating complementary strengths to human reviews. The work also addresses ethical considerations and advocates a human-in-the-loop deployment with open-sourcing to promote transparency and responsible use.

Abstract

As the volume of scientific submissions continues to grow rapidly, traditional peer review systems are facing unprecedented scalability pressures, highlighting the urgent need for automated reviewing methods that are both scalable and reliable. Existing supervised fine-tuning approaches based on real review data are fundamentally constrained by single-source of data as well as the inherent subjectivity and inconsistency of human reviews, limiting their ability to support high-quality automated reviewers. To address these issues, we propose EchoReview, a citation-context-driven data synthesis framework that systematically mines implicit collective evaluative signals from academic citations and transforms scientific community's long-term judgments into structured review-style data. Based on this pipeline, we construct EchoReview-16K, the first large-scale, cross-conference, and cross-year citation-driven review dataset, and train an automated reviewer, EchoReviewer-7B. Experimental results demonstrate that EchoReviewer-7B can achieve significant and stable improvements on core review dimensions such as evidence support and review comprehensiveness, validating citation context as a robust and effective data paradigm for reliable automated peer review.

Figures (4)

• Figure 1: Overview of the EchoReview automated data construction pipeline.
• Figure 3: Performance comparison of reviewer models on multiple quality dimensions.
• Figure 4: Pairwise comparison results between TimeFilter and NoTimeFilter across different review dimensions.
• Figure 5: Comparison of review quality across five evaluation dimensions between DeepReviewer and EchoReviewer after incremental fine-tuning.