ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review

TL;DR

ScholarPeer introduces a context-aware, multi-agent framework for automated peer review that grounds critiques in live literature. By separating knowledge acquisition, active verification via a multi-aspect Q&A engine, and venue-specific synthesis, the system constructs a dynamic external context to assess novelty and methodological soundness. Empirical evaluation on DeepReview-13K shows high alignment with human judgments, strong side-by-side win rates, and favorable novelty and diversity metrics, while ablations demonstrate the critical role of active verification and contextualization. The work emphasizes the importance of grounding AI-generated reviews in up-to-date evidence and discusses ethical considerations such as automation bias and potential homogenization. Overall, ScholarPeer advances automated reviewing by combining live web-scale retrieval with a structured multi-agent reasoning process, achieving a new state-of-the-art in several evaluation metrics while outlining practical trade-offs in latency and cost.

Abstract

Automated peer review has evolved from simple text classification to structured feedback generation. However, current state-of-the-art systems still struggle with "surface-level" critiques: they excel at summarizing content but often fail to accurately assess novelty and significance or identify deep methodological flaws because they evaluate papers in a vacuum, lacking the external context a human expert possesses. In this paper, we introduce ScholarPeer, a search-enabled multi-agent framework designed to emulate the cognitive processes of a senior researcher. ScholarPeer employs a dual-stream process of context acquisition and active verification. It dynamically constructs a domain narrative using a historian agent, identifies missing comparisons via a baseline scout, and verifies claims through a multi-aspect Q&A engine, grounding the critique in live web-scale literature. We evaluate ScholarPeer on DeepReview-13K and the results demonstrate that ScholarPeer achieves significant win-rates against state-of-the-art approaches in side-by-side evaluations and reduces the gap to human-level diversity.

Figures (5)

• Figure 1: Comparative evaluation of ScholarPeer against existing frameworks on DeepReview-13K. (Left) Win rate of ScholarPeer against review fine-tuned models and agentic baselines. (Middle) Average H-Max score (higher the better) of reviews generated by various frameworks (best human review is considered as 5). (Right) Spearman correlation of scores generated by review frameworks with ground-truth human rankings. We use Gemini 3 Pro as the backbone model for ScholarPeer and baseline agentic frameworks. We use Claude Sonnet 4.5 as the LLM-judge. These results show that our proposed framework consistently outperforms baselines across various metrics.
• Figure 2: The ScholarPeer framework: Given an input paper, the framework employs a dual-stream information retrieval process. The knowledge acquisition and contextualization module uses summarizer, search-enabled literature review, historian and baseline scout agents to compress internal and external information. These inputs feed into the multi-aspect Q&A engine, which generates and answers probing questions regarding the novelty and technical soundness. Finally, the review generator utilizes these inputs and conference-specific review guidelines to generate the final review.
• Figure 3: Alignment between LLM and human judges. We observe strong agreement in (a) side-by-side (SxS) win rates and (b) H-Max score. Panel (c) illustrates that $>25\%$ of the reviews generated by ScholarPeer achieve a score of 8 or higher.
• Figure 4: Qualitative analysis of ScholarPeer vs. top baselines. We summarize the key comparative advantages and disadvantages derived from the reasoning traces of the LLM Judge. ScholarPeer dominates in external verification (SOTA checking) and contextual depth, while AI Scientist v2 remains competitive on internal consistency checks.
• Figure 5: Impact of search rounds ($k$) on overall review quality. We observe diminishing returns after $k=3$, where retrieving tangentially related papers begins to dilute the context window.