HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences

TL;DR

This study defines HalluCitation as non-existent references found in NLP conference papers and systematically measures their prevalence across ACL, NAACL, and EMNLP papers from 2024–2025. Using OCR-based citation extraction, fuzzy title matching, and conservative manual verification, the authors identify a sharp rise in HalluCited papers in 2025, with EMNLP 2025 contributing the majority. They show that papers with multiple HalluCitation candidates are more likely to contain true HalluCitations, while many HalluCited papers contain only one or two such citations, making detection challenging without automation. The work highlights implications for peer review, calls for automatic verification tools and traceability, and argues for a collaborative approach to preserve conference credibility without unduly penalizing authors. It also discusses database contamination as a non-malicious contributing factor and proposes concrete policy and tooling changes to enhance citation reliability in future scholarly communication.

Abstract

Recently, we have often observed hallucinated citations or references that do not correspond to any existing work in papers under review, preprints, or published papers. Such hallucinated citations pose a serious concern to scientific reliability. When they appear in accepted papers, they may also negatively affect the credibility of conferences. In this study, we refer to hallucinated citations as "HalluCitation" and systematically investigate their prevalence and impact. We analyze all papers published at ACL, NAACL, and EMNLP in 2024 and 2025, including main conference, Findings, and workshop papers. Our analysis reveals that nearly 300 papers contain at least one HalluCitation, most of which were published in 2025. Notably, half of these papers were identified at EMNLP 2025, the most recent conference, indicating that this issue is rapidly increasing. Moreover, more than 100 such papers were accepted as main conference and Findings papers at EMNLP 2025, affecting the credibility.

Figures (5)

• Figure 1: Examples of incorrect reference information. Some references include incorrect arXiv IDs or contain uncertain phrases such as "Anticipated for". Even references that appear plausible may contain HalluCitations, e.g., incorrect link information or references to pages where the cited paper does not exist, as in the last case.
• Figure 2: Our analysis method. First, we extract citations from the collected PDFs. Next, we attempt to identify the corresponding papers in a reference paper database using fuzzy matching on citation titles, and flag citations for which no matching paper can be found as HalluCitation candidates. Finally, we manually verify the existence of each candidate by referring to the original PDFs and conducting web searches. If it cannot be objectively identified, we mark it as a HalluCitation.
• Figure 3: Number and proportion of HalluCited papers by area. We report areas with three or more papers. Area names are abbreviated using the first few words.
• Figure 4: Word clouds based on TF-IDF differences between HalluCited papers and general papers.
• Figure 5: Examples of incorrect database entries from Google Scholar and Semantic Scholar. Case 1 shows an incorrect title, Case 2 refers to a non-existent paper, and Case 3 contains inaccurate citation information.