Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation

Weijia Zhang, Mohammad Aliannejadi, Jiahuan Pei, Yifei Yuan, Jia-Hong Huang, Evangelos Kanoulas

TL;DR

This paper addresses the challenge of evaluating fine-grained citation support in retrieval-augmented LLMs, where existing faithfulness metrics typically operate in a binary setting. It proposes a comparative evaluation framework with three protocols—correlation analysis, three-way classification, and retrieval evaluation—to align metric scores with human judgments across full, partial, and no support levels. Across seven faithfulness metrics (both similarity-based and entailment-based), the study finds no single metric consistently dominates any protocol; partial support remains particularly difficult to detect, and retrieval scenarios favor similarity-based metrics due to robustness to noise. The work provides practical guidance for metric development, including collecting fine-grained annotations and applying contrastive learning, to improve the reliability of automated citation evaluation in real-world, post-hoc retrieval contexts.

Abstract

Large language models (LLMs) often generate content with unsupported or unverifiable content, known as "hallucinations." To address this, retrieval-augmented LLMs are employed to include citations in their content, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies tackle this challenge by leveraging faithfulness metrics to estimate citation support automatically. However, they limit this citation support estimation to a binary classification scenario, neglecting fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results indicate no single metric consistently excels across all evaluations, highlighting the complexity of accurately evaluating fine-grained support levels. Particularly, we find that the best-performing metrics struggle to distinguish partial support from full or no support. Based on these findings, we provide practical recommendations for developing more effective metrics.

A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation

TL;DR

This paper addresses the challenge of evaluating fine-grained citation support in retrieval-augmented LLMs, where existing faithfulness metrics typically operate in a binary setting. It proposes a comparative evaluation framework with three protocols—correlation analysis, three-way classification, and retrieval evaluation—to align metric scores with human judgments across full, partial, and no support levels. Across seven faithfulness metrics (both similarity-based and entailment-based), the study finds no single metric consistently dominates any protocol; partial support remains particularly difficult to detect, and retrieval scenarios favor similarity-based metrics due to robustness to noise. The work provides practical guidance for metric development, including collecting fine-grained annotations and applying contrastive learning, to improve the reliability of automated citation evaluation in real-world, post-hoc retrieval contexts.

Abstract

Large language models (LLMs) often generate content with unsupported or unverifiable content, known as "hallucinations." To address this, retrieval-augmented LLMs are employed to include citations in their content, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies tackle this challenge by leveraging faithfulness metrics to estimate citation support automatically. However, they limit this citation support estimation to a binary classification scenario, neglecting fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results indicate no single metric consistently excels across all evaluations, highlighting the complexity of accurately evaluating fine-grained support levels. Particularly, we find that the best-performing metrics struggle to distinguish partial support from full or no support. Based on these findings, we provide practical recommendations for developing more effective metrics.
Paper Structure (24 sections, 2 figures, 4 tables)

This paper contains 24 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Evaluation Framework
  3. Task Formulation
  4. Evaluation Protocols
  5. Correlation Analysis
  6. Classification Evaluation
  7. Retrieval Evaluation
  8. Faithfulness Metrics
  9. Similarity-Based Metrics
  10. Entailment-Based Metrics
  11. Experiments
  12. Datasets
  13. Data Statistics
  14. Data Processing
  15. Meta-Evaluation
  16. ...and 9 more sections

Figures (2)

  • Figure 1: An example of partial support in citation evaluation. A retrieval-augmented LLM generates a response that includes citations based on a given user query. The human assessor annotates that the first citation partially supports the associated statement. Inconsistent metric scores are observed when assessing the statement using three distinct faithfulness metrics.
  • Figure 2: Our proposed comparative evaluation framework. A faithfulness metric assigns scores to given statements and their corresponding citations. Subsequently, our framework comprehensively assesses the alignment between these metric scores and human judgments by employing correlation analysis, classification, and retrieval evaluation.