Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Document Attribution: Examining Citation Relationships using Large Language Models

Vipula Rawte, Ryan A. Rossi, Franck Dernoncourt, Nedim Lipka

TL;DR

This paper tackles trust and provenance in document-grounded LLM outputs by reframing attribution as a binary task via zero-shot textual entailment, and by exploring whether attention signals can aid attribution with a lightweight model. It shows that a zero-shot prompt using textual entailment with Flan-UL2 achieves strong ID and OOD performance on AttributionBench, while attention-based attribution with Flan-T5-Small provides mixed but generally favorable results across layers. The work demonstrates that simple, prompt-based attribution can rival more involved baselines and highlights attention as a potential path for improvement, though practical gains are tempered by computational constraints. Overall, it contributes to interpretable provenance verification for document-grounded NLP and points to scalable directions for future enhancement through fine-tuning and deeper analysis of attention mechanisms.

Abstract

As Large Language Models (LLMs) are increasingly applied to document-based tasks - such as document summarization, question answering, and information extraction - where user requirements focus on retrieving information from provided documents rather than relying on the model's parametric knowledge, ensuring the trustworthiness and interpretability of these systems has become a critical concern. A central approach to addressing this challenge is attribution, which involves tracing the generated outputs back to their source documents. However, since LLMs can produce inaccurate or imprecise responses, it is crucial to assess the reliability of these citations. To tackle this, our work proposes two techniques. (1) A zero-shot approach that frames attribution as a straightforward textual entailment task. Our method using flan-ul2 demonstrates an improvement of 0.27% and 2.4% over the best baseline of ID and OOD sets of AttributionBench, respectively. (2) We also explore the role of the attention mechanism in enhancing the attribution process. Using a smaller LLM, flan-t5-small, the F1 scores outperform the baseline across almost all layers except layer 4 and layers 8 through 11.

Document Attribution: Examining Citation Relationships using Large Language Models

TL;DR

This paper tackles trust and provenance in document-grounded LLM outputs by reframing attribution as a binary task via zero-shot textual entailment, and by exploring whether attention signals can aid attribution with a lightweight model. It shows that a zero-shot prompt using textual entailment with Flan-UL2 achieves strong ID and OOD performance on AttributionBench, while attention-based attribution with Flan-T5-Small provides mixed but generally favorable results across layers. The work demonstrates that simple, prompt-based attribution can rival more involved baselines and highlights attention as a potential path for improvement, though practical gains are tempered by computational constraints. Overall, it contributes to interpretable provenance verification for document-grounded NLP and points to scalable directions for future enhancement through fine-tuning and deeper analysis of attention mechanisms.

Abstract

As Large Language Models (LLMs) are increasingly applied to document-based tasks - such as document summarization, question answering, and information extraction - where user requirements focus on retrieving information from provided documents rather than relying on the model's parametric knowledge, ensuring the trustworthiness and interpretability of these systems has become a critical concern. A central approach to addressing this challenge is attribution, which involves tracing the generated outputs back to their source documents. However, since LLMs can produce inaccurate or imprecise responses, it is crucial to assess the reliability of these citations. To tackle this, our work proposes two techniques. (1) A zero-shot approach that frames attribution as a straightforward textual entailment task. Our method using flan-ul2 demonstrates an improvement of 0.27% and 2.4% over the best baseline of ID and OOD sets of AttributionBench, respectively. (2) We also explore the role of the attention mechanism in enhancing the attribution process. Using a smaller LLM, flan-t5-small, the F1 scores outperform the baseline across almost all layers except layer 4 and layers 8 through 11.
Paper Structure (14 sections, 1 equation, 1 figure, 3 tables)

This paper contains 14 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method and Experimental Setup
  4. Zero-shot Textual Entailment
  5. Why zero-shot Textual Entailment?
  6. Attention-based attribution
  7. Results and Analysis
  8. Evaluation Metrics
  9. Zero-shot
  10. Using Attention layers
  11. Conclusion and Future Work
  12. Limitations
  13. Limitation 1:
  14. Limitation 2:

Figures (1)

  • Figure 1: For our zero-shot experiments, we used this prompt template to query the LLM for determining whether the REFERENCE entails the CLAIM.