Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HART: Data-Driven Hallucination Attribution and Evidence-Based Tracing for Large Language Models

Shize Liang, Hongzhi Wang

TL;DR

HART formalizes hallucination tracing as a structured modeling task comprising four stages: span localization, mechanism attribution, evidence retrieval, and causal tracing, and develops the first structured dataset tailored for hallucination tracing, which demonstrates that HART substantially outperforms strong retrieval baselines, including BM25 and DPR.

Abstract

Large language models (LLMs) have demonstrated remarkable performance in text generation and knowledge-intensive question answering. Nevertheless, they are prone to producing hallucinated content, which severely undermines their reliability in high-stakes application domains. Existing hallucination attribution approaches, based on either external knowledge retrieval or internal model mechanisms, primarily focus on semantic similarity matching or representation-level discrimination. As a result, they have difficulty establishing structured correspondences at the span level between hallucination types, underlying error generation mechanisms, and external factual evidence, thereby limiting the interpretability of hallucinated fragments and the traceability of supporting or opposing evidence. To address these limitations, we propose HART, a fine-grained hallucination attribution and evidence retrieval framework for large language models. HART formalizes hallucination tracing as a structured modeling task comprising four stages: span localization, mechanism attribution, evidence retrieval, and causal tracing. Based upon this formulation, we develop the first structured dataset tailored for hallucination tracing, in which hallucination types, error mechanisms, and sets of counterfactual evidence are jointly annotated to enable causal-level interpretability evaluation. Experimental results on the proposed dataset demonstrate that HART substantially outperforms strong retrieval baselines, including BM25 and DPR, validating the effectiveness and generalization capability of the proposed tracing paradigm for hallucination analysis and evidence alignment.

HART: Data-Driven Hallucination Attribution and Evidence-Based Tracing for Large Language Models

TL;DR

HART formalizes hallucination tracing as a structured modeling task comprising four stages: span localization, mechanism attribution, evidence retrieval, and causal tracing, and develops the first structured dataset tailored for hallucination tracing, which demonstrates that HART substantially outperforms strong retrieval baselines, including BM25 and DPR.

Abstract

Large language models (LLMs) have demonstrated remarkable performance in text generation and knowledge-intensive question answering. Nevertheless, they are prone to producing hallucinated content, which severely undermines their reliability in high-stakes application domains. Existing hallucination attribution approaches, based on either external knowledge retrieval or internal model mechanisms, primarily focus on semantic similarity matching or representation-level discrimination. As a result, they have difficulty establishing structured correspondences at the span level between hallucination types, underlying error generation mechanisms, and external factual evidence, thereby limiting the interpretability of hallucinated fragments and the traceability of supporting or opposing evidence. To address these limitations, we propose HART, a fine-grained hallucination attribution and evidence retrieval framework for large language models. HART formalizes hallucination tracing as a structured modeling task comprising four stages: span localization, mechanism attribution, evidence retrieval, and causal tracing. Based upon this formulation, we develop the first structured dataset tailored for hallucination tracing, in which hallucination types, error mechanisms, and sets of counterfactual evidence are jointly annotated to enable causal-level interpretability evaluation. Experimental results on the proposed dataset demonstrate that HART substantially outperforms strong retrieval baselines, including BM25 and DPR, validating the effectiveness and generalization capability of the proposed tracing paradigm for hallucination analysis and evidence alignment.
Paper Structure (24 sections, 15 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 24 sections, 15 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG
  4. Model internal mechanism traceability
  5. Methodlogy
  6. Overview
  7. Datasets Construction
  8. Retrieval
  9. Evidence-level Document Corpus Construction
  10. Semantic Vector Encoding
  11. Vector Indexing and Retrieval
  12. Evidence Reranking and Semantic Refinement
  13. Hit Determination and Theoretical Consistency
  14. Evidence Tracing
  15. Problem Definition
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The construction of the hallucination tracing dataset employs a labeling framework combining large language model assistance with human supervision.
  • Figure 2: Evidence Retrieval Process: Leveraging semantic embeddings, vector-based indexing, and a Cross-Encoder fine-ranking mechanism, this process aligns hallucinated fragments with external factual evidence and retrieves the most relevant supporting information.
  • Figure 3: Span-level hallucination tracing workflow: HART integrates hallucination type classification, error mechanism attribution, and evidence retrieval to systematically model and trace the external evidence supporting each hallucinated segment in model-generated text.