Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection

Federica Gamba, Aman Sinha, Timothee Mickus, Raul Vazquez, Patanjali Bhamidipati, Claudio Savelli, Ahana Chattopadhyay, Laura A. Zanella, Yash Kankanampati, Binesh Arakkal Remesh, Aryan Ashok Chandramania, Rohit Agarwal, Chuyuan Li, Ioana Buhnila, Radhika Mamidi

TL;DR

This work introduces CAP, a multilingual dataset for scientific hallucination detection that spans nine languages and 900 questions grounded in ACL publications. It provides 7000+ LLM-generated answers from 16 models, annotated for factuality and fluency along with token sequences and logits, enabling both cross-lingual evaluation and analysis of input context effects. The authors benchmark six hallucination detectors, revealing that existing tools struggle in multilingual scientific QA and that factors such as question type and context complexity drive hallucination rates more than citation popularity. CAP thus offers a challenging benchmark for developing robust, multilingual, factually grounded scientific text generation.

Abstract

We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages (English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and Telugu). The dataset comprises 900 curated scientific questions and over 7000 LLM-generated answers from 16 publicly available models, provided as question-answer pairs along with token sequences and corresponding logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the development of more reliable scientific NLP systems.

Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection

TL;DR

This work introduces CAP, a multilingual dataset for scientific hallucination detection that spans nine languages and 900 questions grounded in ACL publications. It provides 7000+ LLM-generated answers from 16 models, annotated for factuality and fluency along with token sequences and logits, enabling both cross-lingual evaluation and analysis of input context effects. The authors benchmark six hallucination detectors, revealing that existing tools struggle in multilingual scientific QA and that factors such as question type and context complexity drive hallucination rates more than citation popularity. CAP thus offers a challenging benchmark for developing robust, multilingual, factually grounded scientific text generation.

Abstract

We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages (English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and Telugu). The dataset comprises 900 curated scientific questions and over 7000 LLM-generated answers from 16 publicly available models, provided as question-answer pairs along with token sequences and corresponding logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the development of more reliable scientific NLP systems.
Paper Structure (25 sections, 7 figures, 5 tables)

This paper contains 25 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. CAP Dataset
  4. Overview
  5. CAP Creation Workflow
  6. Scientific Paper Collection:
  7. Question Creation:
  8. Response Generation using LLMs:
  9. Annotation Details
  10. Human Annotators:
  11. Prompt Design:
  12. Annotation Procedure:
  13. CAP Statistics
  14. Characteristics of CAP Dataset
  15. Effects of Citation Counts
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Distribution of publicly available text generation models on HuggingFace (HF) website as of October 22nd, 2025.
  • Figure 2: Example instance extracted from CAP Malayalam. The question roughly translates to "Which Indian languages are used in this test set?". It provides all the metadata regarding the creation of this example including information related to the question and the response generated.
  • Figure 3: Standardized template used for response generation in English, analogous templates are used for other languages.
  • Figure 4: Label distribution per language.
  • Figure 5: Mean citation per language.
  • ...and 2 more figures