ANAH: Analytical Annotation of Hallucinations in Large Language Models

TL;DR

ANAH presents a large-scale bilingual (English-Chinese) benchmark for fine-grained sentence-level analytic annotation of hallucinations in knowledge-based generative QA. It outlines a four-stage dataset construction pipeline, introduces both generative and discriminative hallucination annotators trained on the ANAH data, and demonstrates that a sufficiently large generative annotator can rival GPT-4 in accuracy while remaining cost-efficient. The work also analyzes phenomena such as the accumulation of hallucinations and the generalization behavior of annotators across topics and questions. Together, ANAH enables precise grounding and correction of hallucinations, with clear pathways to integration into mitigation pipelines and future scaling to broader domains. The study provides a comprehensive framework for evaluating and improving fine-grained hallucination detection in multilingual contexts and sets benchmarks for cross-model annotation quality and generalization.

Abstract

Reducing the `$\textit{hallucination}$' problem of Large Language Models (LLMs) is crucial for their wide applications. A comprehensive and fine-grained measurement of the hallucination is the first key step for the governance of this issue but is under-explored in the community. Thus, we present $\textbf{ANAH}$, a bilingual dataset that offers $\textbf{AN}$alytical $\textbf{A}$nnotation of $\textbf{H}$allucinations in LLMs within Generative Question Answering. Each answer sentence in our dataset undergoes rigorous annotation, involving the retrieval of a reference fragment, the judgment of the hallucination type, and the correction of hallucinated content. ANAH consists of ~12k sentence-level annotations for ~4.3k LLM responses covering over 700 topics, constructed by a human-in-the-loop pipeline. Thanks to the fine granularity of the hallucination annotations, we can quantitatively confirm that the hallucinations of LLMs progressively accumulate in the answer and use ANAH to train and evaluate hallucination annotators. We conduct extensive experiments on studying generative and discriminative annotators and show that, although current open-source LLMs have difficulties in fine-grained hallucination annotation, the generative annotator trained with ANAH can surpass all open-source LLMs and GPT-3.5, obtain performance competitive with GPT-4, and exhibits better generalization ability on unseen questions.

Figures (17)

• Figure 1: An example of ANAH for sentence-level hallucination annotation. Each sentence in a generated answer is annotated in fine-grained with Reference Fragment, Hallucination Type, and Correction. The hallucinated and supported content are highlighted in orange and blue, respectively.
• Figure 2: The overview of dataset establishment, comprising (a) Topic Selection and Reference Retrieval, (b) Question Generation and Selection, (c) Answer Generation, and (d) Fine-grained Hallucination Annotation.
• Figure 3: The topic distribution by chart of (a) categories (inner) and domains (outer), and (b) word cloud.
• Figure 4: Hallucination Type Confusion Matrices for InternLM2-20B-based generative annotator (a) and discriminative annotator (b).
• Figure A1: Prompts for Reference Retrieval.