LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv

TL;DR

The paper investigates arXiv's 2025 CS ban on unpublished review papers by quantifying LLM-generated content in review and non-review papers from 2020–2025 using two detectors (Alpha and Pangram) and a review-vs-non-review classifier. It finds that while review papers show higher proportions of LLM-generated content, non-review papers contribute far more to the total AI-generated literature, and post-ChatGPT adoption surged across all domains. Subfield variation is substantial, with Computers & Society particularly affected, and disciplinary context heavily modulates policy impact. The study advocates evidence-based, discipline-aware moderation rather than blanket bans and provides open-source code to enable replication and ongoing monitoring.

Abstract

ArXiv recently prohibited the upload of unpublished review papers to its servers in the Computer Science domain, citing a high prevalence of LLM-generated content in these categories. However, this decision was not accompanied by quantitative evidence. In this work, we investigate this claim by measuring the proportion of LLM-generated content in review vs. non-review research papers in recent years. Using two high-quality detection methods, we find a substantial increase in LLM-generated content across both review and non-review papers, with a higher prevalence in review papers. However, when considering the number of LLM-generated papers published in each category, the estimates of non-review LLM-generated papers are almost six times higher. Furthermore, we find that this policy will affect papers in certain domains far more than others, with the CS subdiscipline Computers & Society potentially facing cuts of 50%. Our analysis provides an evidence-based framework for evaluating such policy decisions, and we release our code to facilitate future investigations at: https://github.com/yanaiela/llm-review-arxiv.

Figures (11)

• Figure 1: LLM-generated paper estimates using Pangram detection for review vs. non-review papers. While review papers show higher percentages of LLM-generated content (in parentheses), the absolute number of non-review papers detected as LLM-generated is substantially larger in all domains, suggesting that restricting review papers alone may not address the core concern about LLM-generated content on arXiv.
• Figure 2: LLM-generated papers estimates using (a) Alpha estimates and (b) Pangram detection ratios on the arxiv-domains dataset. Both methods reveal temporal patterns in LLM-adoption across domains, and years.
• Figure 3: LLM-generated papers estimates by subcategories and paper type in cs-subcategories using (a) Alpha estimates and (b) Pangram detection ratios. The review versus non-review paper gap (or lack thereof) varies considerably across subcategories, suggesting that field-specific norms influence AI adoption patterns.
• Figure 4: LLM-generated papers estimates by year in CS (arxiv-domains) using Pangram, when considering the entire text. While the percentages drop by about half compared to the abstract estimates, the yearly upward trend, as well as the review vs. non-review gap persist.
• Figure 5: LLM-generated papers estimates by domain and paper type on the arxiv-domains dataset using (a) Alpha estimates and (b) Pangram detection ratios. Review papers consistently show higher LLM-generated ratios for CS and Physics, and mixed results for Statistics and Math depending on the method. Results are averaged across the post-LLM years (2023-2025).