Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation

Yee Man Choi, Xuehang Guo, Yi R. Fung, Qingyun Wang

TL;DR

CiteGuard addresses the problem of faithful citation attribution in LLM-generated scientific writing by reframing citation evaluation as attribution alignment and introducing a retrieval-augmented agent. The method extends prior work with a rich set of retrieval actions that query and verify candidate references across a Semantic Scholar corpus, enabling iterative generation of one or more appropriate citations per excerpt. Empirical results on the CiteME benchmark show that CiteGuard outperforms LLM-as-a-Judge-based baselines by 12.3% and reaches up to 65.4% accuracy with open models, approaching human performance at 69.7%, while also producing high-quality alternative citations validated by human annotators. The work highlights the tradeoffs between retrieval depth and token cost, demonstrates model-agnostic effectiveness across reasoning and non-reasoning models, and discusses limitations and avenues for future work, including expanding data sources beyond Semantic Scholar and analyzing performance on smaller open models.

Abstract

Large Language Models (LLMs) have emerged as promising assistants for scientific writing. However, there have been concerns regarding the quality and reliability of the generated text, one of which is the citation accuracy and faithfulness. While most recent work relies on methods such as LLM-as-a-Judge, the reliability of LLM-as-a-Judge alone is also in doubt. In this work, we reframe citation evaluation as a problem of citation attribution alignment, which is assessing whether LLM-generated citations match those a human author would include for the same text. We propose CiteGuard, a retrieval-aware agent framework designed to provide more faithful grounding for citation validation. CiteGuard improves the prior baseline by 12.3%, and achieves up to 65.4% accuracy on the CiteME benchmark, on par with human-level performance (69.7%). It also enables the identification of alternative but valid citations.

CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation

TL;DR

CiteGuard addresses the problem of faithful citation attribution in LLM-generated scientific writing by reframing citation evaluation as attribution alignment and introducing a retrieval-augmented agent. The method extends prior work with a rich set of retrieval actions that query and verify candidate references across a Semantic Scholar corpus, enabling iterative generation of one or more appropriate citations per excerpt. Empirical results on the CiteME benchmark show that CiteGuard outperforms LLM-as-a-Judge-based baselines by 12.3% and reaches up to 65.4% accuracy with open models, approaching human performance at 69.7%, while also producing high-quality alternative citations validated by human annotators. The work highlights the tradeoffs between retrieval depth and token cost, demonstrates model-agnostic effectiveness across reasoning and non-reasoning models, and discusses limitations and avenues for future work, including expanding data sources beyond Semantic Scholar and analyzing performance on smaller open models.

Abstract

Large Language Models (LLMs) have emerged as promising assistants for scientific writing. However, there have been concerns regarding the quality and reliability of the generated text, one of which is the citation accuracy and faithfulness. While most recent work relies on methods such as LLM-as-a-Judge, the reliability of LLM-as-a-Judge alone is also in doubt. In this work, we reframe citation evaluation as a problem of citation attribution alignment, which is assessing whether LLM-generated citations match those a human author would include for the same text. We propose CiteGuard, a retrieval-aware agent framework designed to provide more faithful grounding for citation validation. CiteGuard improves the prior baseline by 12.3%, and achieves up to 65.4% accuracy on the CiteME benchmark, on par with human-level performance (69.7%). It also enables the identification of alternative but valid citations.
Paper Structure (34 sections, 2 equations, 16 figures, 5 tables)

This paper contains 34 sections, 2 equations, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. CiteGuard
  3. Problem Formulation
  4. Reference Retrieval
  5. Experiments
  6. CiteGuard Accurately Grounds Scientific Claims Through Enhanced Actions
  7. CiteGuard Effectively Suggests Alternative Citations
  8. Analysis
  9. Retrieval vs Long-Context
  10. Reasoning vs Non-Reasoning Models
  11. CiteGuard vs Paper Finders
  12. Conclusion
  13. Related Work
  14. Retrieval-Augmented Generation and LLMs for Scientific Research
  15. Citation Suggestion
  16. ...and 19 more sections

Figures (16)

  • Figure 1: CiteGuard succeeds through expanded retrieval actions, where CiteAgent press2024citeme fails due to OpenPDF access error.
  • Figure 2: CiteGuard takes an excerpt and performs searches to find a paper that best matches the missing citation.
  • Figure 3: Retrieving Actions. We define six retrieval actions to ensure the efficiency and accuracy of CiteGuard.
  • Figure 4: CiteGuard System Prompt
  • Figure 5: CiteGuard ask_for_more_context Prompt
  • ...and 11 more figures