Detecting AI-Generated Content in Academic Peer Reviews
Siyuan Shen, Kai Wang
TL;DR
This study addresses the emergence of AI-generated content in academic peer reviews by training a Longformer+LoRA detector on 2021 reviews and applying it to 2022–2025 reviews from ICLR and Nature Communications to assess temporal generalization. The approach enables a realistic, year-based analysis across two distinct review formats, coupled with synthetic-data generation (via DeepSeek Reasoner) to train the detector without future disclosures. Findings show near-zero AI-generated classifications before 2022, with AI-detected reviews rising to about $20 ext{ extcurrency}$ in ICLR and $12 ext{ extcurrency}$ in NC by 2025, including a pronounced quarterly rise in late 2024 for NC. The work highlights implications for transparency, policy, and further cross-domain validation, while candidly noting limitations such as the binary labeling assumption and potential overfitting to synthetic data.
Abstract
The growing availability of large language models (LLMs) has raised questions about their role in academic peer review. This study examines the temporal emergence of AI-generated content in peer reviews by applying a detection model trained on historical reviews to later review cycles at International Conference on Learning Representations (ICLR) and Nature Communications (NC). We observe minimal detection of AI-generated content before 2022, followed by a substantial increase through 2025, with approximately 20% of ICLR reviews and 12% of Nature Communications reviews classified as AI-generated in 2025. The most pronounced growth of AI-generated reviews in NC occurs between the third and fourth quarter of 2024. Together, these findings provide suggestive evidence of a rapidly increasing presence of AI-assisted content in peer review and highlight the need for further study of its implications for scholarly evaluation.