Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics

Weijia Zhang, Mohammad Aliannejadi, Yifei Yuan, Jiahuan Pei, Jia-Hong Huang, Evangelos Kanoulas

TL;DR

The paper tackles the problem of evaluating fine-grained citation support in generated text, where citations may fully, partially, or not support a statement. It proposes a comparative evaluation framework comprising correlation analysis, classification evaluation, and retrieval evaluation to assess faithfulness metrics across three support levels. Experimental results on GenSearch show that no single metric consistently outperforms others across all protocols; similarity-based metrics often fare better in retrieval, while entailment-based metrics can struggle with noise, and LLM-based metrics underperform overall. The work highlights protocol complementarity and offers practical recommendations—such as richer fine-grained training data and contrastive learning—to drive the development of more robust, explainable, and effective citation-faithfulness metrics with implications for grounded generation systems.

Abstract

Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations." To mitigate this, retrieval-augmented LLMs incorporate citations, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies use faithfulness metrics to estimate citation support automatically but are limited to binary classification, overlooking fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results show no single metric consistently excels across all evaluations, revealing the complexity of assessing fine-grained support. Based on the findings, we provide practical recommendations for developing more effective metrics.

Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics

TL;DR

The paper tackles the problem of evaluating fine-grained citation support in generated text, where citations may fully, partially, or not support a statement. It proposes a comparative evaluation framework comprising correlation analysis, classification evaluation, and retrieval evaluation to assess faithfulness metrics across three support levels. Experimental results on GenSearch show that no single metric consistently outperforms others across all protocols; similarity-based metrics often fare better in retrieval, while entailment-based metrics can struggle with noise, and LLM-based metrics underperform overall. The work highlights protocol complementarity and offers practical recommendations—such as richer fine-grained training data and contrastive learning—to drive the development of more robust, explainable, and effective citation-faithfulness metrics with implications for grounded generation systems.

Abstract

Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations." To mitigate this, retrieval-augmented LLMs incorporate citations, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies use faithfulness metrics to estimate citation support automatically but are limited to binary classification, overlooking fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results show no single metric consistently excels across all evaluations, revealing the complexity of assessing fine-grained support. Based on the findings, we provide practical recommendations for developing more effective metrics.
Paper Structure (27 sections, 3 figures, 5 tables)

This paper contains 27 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Faithfulness Evaluation Metrics
  4. Citation Evaluation
  5. Evaluation Framework
  6. Task Formulation
  7. Evaluation Protocols
  8. Correlation Analysis
  9. Classification Evaluation
  10. Retrieval Evaluation
  11. Faithfulness Metrics
  12. Similarity-Based Metrics
  13. Entailment-Based Metrics
  14. LLM-Based Metrics
  15. Experiments
  16. ...and 12 more sections

Figures (3)

  • Figure 1: An example of partial support in citation evaluation. Inconsistent metric scores are observed when assessing the statement with three faithfulness metrics.
  • Figure 2: The overview of the proposed comparative evaluation framework. A faithfulness metric assigns scores to given statements and their corresponding citations. Subsequently, our framework comprehensively assesses the alignment between these metric scores and human judgments by employing correlation analysis, classification, and retrieval evaluation.
  • Figure 3: Retrieval performance of faithfulness metrics regarding NDCG@n scores on the GenSearch dataset. Note that we assign relevance labels $2$, $1$, and $0$ to full, partial, and no support, respectively (shown in the color).