Citation Failure: Definition, Analysis and Efficient Mitigation

TL;DR

This work addresses citation failure in LLM-based retrieval-augmented generation, where a correct answer may be supported by incomplete evidence. It introduces CiteControl, a controlled benchmark that isolates how the relation between a response and its evidence affects citation quality, enabling separation of response failure from citation failure. Building on this, the authors propose Citentation (Citation framework), which unifies generation-based, attention-based, and retrieval-based citation with an aggregation and decision mechanism to efficiently improve corroborative citations. Empirical results on CiteControl and transfer datasets show that combining methods yields the strongest improvements, particularly in complex reasoning tasks, and demonstrate effective citation enhancement without task-specific in-domain training. These contributions advance robust, verifiable LLM responses in retrieval-augmented contexts and open avenues for further integration of multiple citation signals and efficient retrieval strategies.

Abstract

Citations from LLM-based RAG systems are supposed to simplify response verification. However, this does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence. In contrast to previous work, we propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible. To address citation failure, this work follows a two-step approach: (1) We study when citation failure occurs and (2) how it can be mitigated. For step 1, we extend prior work by investigating how the relation between response and evidence affects citation quality. We introduce CITECONTROL, a benchmark that systematically varies this relation to analyze failure modes. Experiments show that failures increase with relational complexity and suggest that combining citation methods could improve performance, motivating step 2. To improve LLM citation efficiently, we propose CITENTION, a framework integrating generative, attention-based, and retrieval-based methods. Results demonstrate substantial citation improvements on CITECONTROL and in transfer settings. We make our data and code publicly available.

Figures (7)

• Figure 1: Citation Example: An LLM receives multiple documents and a question. The confusion matrix shows the possible outcomes for generated response and evidence. The response-evidence relation has reasoning type multi-hop. It is explicit for the response and [4], and implicit for the response and [2].
• Figure 2: Evaluation strategy on CiteControl: For R$k$, all predictions are evaluated for evidence recall @$k$, while for R$k^\mathrm{f}$, only predictions with correct responses are evaluated.
• Figure 3: (A) Citation precision decreases with the order of appearance in generation. (B) R$k^\mathrm{f}$ is highest for the evidence document that contains the response (hop 0) and is reduced when going to earlier hops, DRAG and BM25 are notable exceptions (see §\ref{['sec:citation_analysis/results']}). Plots show data for correctly answered instances.
• Figure 4: Overview of Citention. Left: Individual scores for each document are obtained from generation-based, attention-based and retrieval-based methods. Attention scores are averaged over individual heads. Right: Scores from individual methods are summed to obtain a final citation prediction. Attention head weights $\theta$ and method weights $w, b$ are omitted.
• Figure 5: (A) Citention methods improve citation R$k^\mathrm{f}$ scores on CiteControl (§\ref{['sec:attention/citecontrol']}). (B) Citention methods trained on CiteControl improve R$k^\mathrm{f}$ scores on unseen tasks (§\ref{['sec:attention/transfer']}). Bars show proportion of answer statements that are attributable to the evidence (averaged over train datasets). Whiskers show standard deviation. For the unaggregated data see Tab. \ref{['tab:citention_on_transfer']}.