PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

TL;DR

PaperAudit-Bench presents a two-component framework for critical automated peer review: PaperAudit-Dataset provides a long-context, multi-type error corpus by synthetically editing high-quality AI conference papers, and PaperAudit-Review integrates structured error detection with evidence-grounded reviews across three detection depths. Empirical results show substantial model-to-model variability in error detectability, the importance of cross-section context for distributed errors, and that explicitly integrated error detection makes AI reviews more critical and more aligned with human judgments, albeit not perfect. The work also demonstrates the feasibility of training lightweight detectors via SFT and RL on the synthetic dataset, enabling cost-efficient deployment. Overall, PaperAudit-Bench advances controlled evaluation of error-aware automated reviewing and offers pathways for improving reliability and portability of AI-assisted peer review systems.

Abstract

Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper. In this paper, we introduce PaperAudit-Bench, which consists of two components: (1) PaperAudit-Dataset, an error dataset covering both errors identifiable within individual sections and those requiring cross-section reasoning, designed for controlled evaluation under long-context settings; and (2) PaperAudit-Review, an automated review framework that integrates structured error detection with evidence-aware review generation to support critical assessment. Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths, highlighting the difficulty of identifying such errors under long-context settings. Relative to representative automated reviewing baselines, incorporating explicit error detection into the review workflow produces systematically stricter and more discriminative evaluations, demonstrating its suitability for peer review. Finally, we show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.

Figures (8)

• Figure 1: Overview of PaperAudit-Dataset and examples of errors injected through model-driven synthetic editing.
• Figure 2: Statistics of PaperAudit-Dataset. (a) Distribution of research areas covered in PaperAudit-Dataset. (b) Distribution of input token lengths per paper, estimated under the GPT-5 default configuration. (c) Distribution of injected error density, measured as errors per 10K tokens, computed from GPT-5-generated corrupted papers.
• Figure 3: Overview of the PaperAudit-Review Framework. The detection workflow supports three levels of analytical depth, while the review workflow builds upon the detection results to synthesize critical assessments.
• Figure 4: Distribution of injected error types and their locations in the NeurIPS subset of PaperAudit-Dataset. The statistics are averaged over corrupted papers generated by eight synthesis models.
• Figure 5: Detection performance of different models under the Fast mode, evaluated using Macro-F1, Error Coverage (EC), and Finding Precision (FP). Bars show the mean performance with standard deviation across data synthesized by eight synthesis models. Models marked with * denote non-multimodal models where visual content is removed from the input papers. We use Qwen3-235B-A22B instead of Qwen3-VL-235B-A22B, as the latter does not reliably produce outputs in the required parsable format. Unless otherwise specified, all models are evaluated without explicit reasoning mechanisms, except for o4-mini.