AutoRev: Multi-Modal Graph Retrieval for Automated Peer-Review Generation

TL;DR

AutoRev presents a graph-based multi-modal Retrieval-Augmented Generation system for automated peer-review generation. By representing a research paper as a hierarchical graph and using a Graph Attention Network to select salient passages, it reduces input length for large language models and enhances review generation. The approach is trained and evaluated on ICLR 2024 data, demonstrating up to 58.72% improvements over baselines on standard metrics, with supportive human evaluation and LLM-as-a-Judge validation. AutoRev aims to streamline the peer-review workflow, enabling scalable, high-quality feedback to researchers while acknowledging ethical and practical considerations for deployment.

Abstract

Enhancing the quality and efficiency of academic publishing is critical for both authors and reviewers, as research papers are central to scholarly communication and a major source of high-quality content on the web. To support this goal, we propose AutoRev, an automatic peer-review system designed to provide actionable, high-quality feedback to both reviewers and authors. AutoRev leverages a novel Multi-Modal Retrieval-Augmented Generation (RAG) framework that combines textual and graphical representations of academic papers. By modelling documents as graphs, AutoRev effectively retrieves the most pertinent information, significantly reducing the input context length for LLMs and thereby enhancing their review generation capabilities. Experimental results show that AutoRev outperforms state-of-the-art baselines by up to 58.72% and demonstrates competitive performance in human evaluations against ground truth reviews. We envision AutoRev as a powerful tool to streamline the peer-review workflow, alleviating challenges and enabling scalable, high-quality scholarly publishing. By guiding both authors and reviewers, AutoRev has the potential to accelerate the dissemination of quality research on the web at a larger scale. Code will be released upon acceptance.

Figures (6)

• Figure 1: Example document graph with nodes for headings, sub-headings, passages, and sentences. This structure allows AutoRev to capture hierarchical and sequential dependencies across sections.
• Figure 2: The system begins with a parsed document from which DPR extracts passages for GNN training. The GNN learns to identify key passages relevant to review generation. The trained GNN selects important passages without requiring human-generated reviews. These passages, combined with a structured prompt, are input to fine-tune an LLM. The fine-tuned LLM generates the final review efficiently, without external grounding.
• Figure 3: Excerpt of a test sample where passages highlighted in cyan colour refer to relevant passages identified by GNN only and yellow colour refer to relevant passages identified by both GNN and DPR
• Figure 4: This interface component corresponds to the Qualitative Assessment task described in Section \ref{['sec:human-qualitative']}. It marks the beginning of the review presentation. Due to space constraints, only the summary and a subset of the strengths from one review are shown here.
• Figure 5: This interface component corresponds to the task described in Section \ref{['sec:human-qualitative']}. In this part, annotators provide a final rating across the four categories and submit their annotation.