WithdrarXiv: A Large-Scale Dataset for Retraction Study

TL;DR

This work introduces WithdrarXiv, the first large-scale dataset of withdrawn arXiv preprints, comprising over 14,000 papers with retraction comments through September 2024. It derives a 10-category taxonomy of withdrawal reasons from author comments and demonstrates strong zero-shot categorization performance using GPT-4, achieving a weighted F1 around $0.96$. An enriched subset, WithdrarXiv-SciFy, adds parsed full-text PDFs to support scientific feasibility, claim verification, and automated theorem proving. The study also addresses ethical data release by excluding sensitive personal information and implementing gated access and a right-to-be-forgotten policy, highlighting implications for improving scientific integrity and automated verification in rapid-publishing contexts.

Abstract

Retractions play a vital role in maintaining scientific integrity, yet systematic studies of retractions in computer science and other STEM fields remain scarce. We present WithdrarXiv, the first large-scale dataset of withdrawn papers from arXiv, containing over 14,000 papers and their associated retraction comments spanning the repository's entire history through September 2024. Through careful analysis of author comments, we develop a comprehensive taxonomy of retraction reasons, identifying 10 distinct categories ranging from critical errors to policy violations. We demonstrate a simple yet highly accurate zero-shot automatic categorization of retraction reasons, achieving a weighted average F1-score of 0.96. Additionally, we release WithdrarXiv-SciFy, an enriched version including scripts for parsed full-text PDFs, specifically designed to enable research in scientific feasibility studies, claim verification, and automated theorem proving. These findings provide valuable insights for improving scientific quality control and automated verification systems. Finally, and most importantly, we discuss ethical issues and take a number of steps to implement responsible data release while fostering open science in this area.

Figures (5)

• Figure 1: Metadata elements extracted from arXiv abstract pages for building WithdrarXiv
• Figure 2: Confusion matrix for zero-shot categorization as evaluated on human annotations of a 10% stratified sample of the comments (total support=1620)
• Figure 3: Distribution of reasons for paper withdrawals on arXiv. The histogram shows the frequency of different withdrawal categories, ranging from critical errors to policy violations. Each category is represented by a letter (A-J) and color-coded for clarity. Error bars are derived from categorization error rates computed via human evaluation (c.f. Section \ref{['sec:zero-shot-comment-cat']}). For insights from this chart, see Section \ref{['sec:insights-from-cat']}.
• Figure 4: Top 10 arXiv subject categories with their retraction counts. AI topics, such as Computer Vision and Machine Learning (CS.LG), and Quantum Physics occupy the top, with Materials Science at the 10th place. When a preprint is cross-listed in multiple categories, we count it in each applicable category. The annotations in parentheses show retraction rates as percentages for each category.
• Figure 5: Retraction categories for four select subjects -- Computer Vision, Quantum Physics/Computing, Natural Language Processing, and Materials Science (left-to-right, top-to-bottom). Legend: A: 'factual/methodological/other critical errors in manuscript', B: 'subsumed by another publication', C: 'reason not specified', D: 'typos in manuscript', E: 'personal reasons', F: 'administrative or legal issues', G: 'incomplete exposition or more work in progress', H: 'plagiarism', I: 'not novel', J: 'arXiv policy violation'