OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

TL;DR

OpenReviewer presents a specialized long-context 8B-parameter model (Llama-OpenReviewer-8B) fine-tuned on 79K expert ML conference reviews to generate structured, critical pre-submission reviews. It integrates a PDF-to-markdown extraction pipeline (Marker) and a templated, streaming review generation workflow implemented in an open-source demo on HuggingFace Spaces, designed to follow venue guidelines. In a 400-paper evaluation against baselines including GPT-4o and Claude-3.5-Sonnet, OpenReviewer aligns more closely with human reviewers and produces less overly-positive recommendations, indicating the value of domain-specific fine-tuning for review quality. The system is intended as a pre-submission feedback tool to aid authors and complement human peer review, with future work proposed on expanding venues, incorporating citations, and improving evaluation metrics, while addressing ethical considerations.

Abstract

We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Figures (8)

• Figure 1: Annotated screenshot of the https://huggingface.co/spaces/maxidl/openreviewer, with slightly modified layout. 1) Dialogue for uploading a PDF file. 2) Once the user uploads a file, this text field will be populated with the papers' full text in markdown format. The user can choose to edit the text to fix conversion errors. 3) An accordion element to show and optionally edit the review template used for generation. 4) Button to run the review generation and enabled once the paper text field is populated. When clicked, a review is generated in streaming mode and printed below on the fly.
• Figure 2: Preference evaluation using GPT-4o as the annotator, judging which generated review aligns better with a set of human-written reviews.
• Figure 3: System prompt used by OpenReviewer. Fields in {} are placeholders.
• Figure 4: User prompt used by OpenReviewer. Fields in {} are placeholders.
• Figure 5: System prompt for the LLM judge. Fields in {} are placeholders.