C$^2$-Cite: Contextual-Aware Citation Generation for Attributed Large Language Models

TL;DR

This work tackles the challenge of contextually grounding citations in attributed LLMs by introducing C$^2$-Cite, a fine-tuned framework with contextual-aware embeddings and contextual citation alignment. By converting citation markers into semantically rich tokens and training a citation router plus attentive augmentation, the model transforms citations from passive placeholders into active knowledge pointers linked to retrieved sources. On the ALCE benchmark, C$^2$-Cite++ achieves notable gains in both citation quality ($+$5.8% on average) and response correctness ($+$17.4%), while maintaining efficiency, and ablation studies confirm the critical roles of the contextual embedding and attention mechanisms. These results demonstrate a practical impact for building more trustworthy and verifiable LLMs in retrieval-augmented settings, with potential applicability across open-domain QA and scholarly retrieval tasks.

Abstract

The attribution technique enhances the credibility of LLMs by adding citations to the generated sentences, enabling users to trace back to the original sources and verify the reliability of the output. However, existing instruction-tuned attributed LLMs often fail to properly interpret the contextual semantics of citation symbols (e.g., [i]) during text generation. This shortcoming arises from their insufficient awareness of the context information surrounding citation markers, which in turn leads to disjointed references and poor integration of retrieved knowledge into the generated content. To address this issue, we propose a novel \textbf{C}ontextual-aware \textbf{C}itation generation framework (\textbf{C$^2$}-\textbf{Cite}) that explicitly integrates the semantic relationships between citation markers and their referenced content. Specifically, a contextual citation alignment mechanism is adopted: it first encodes the retrieved document contexts into the symbol representation of citations, then aligns the marker numbers by decoding information from a citation router function. This mechanism enables the transformation of citation markers from generic placeholders into active knowledge pointers that link to the referenced source information. Experimental results on the ALCE benchmark across three datasets validate our framework C$^2$-Cite++: it outperforms the SOTA baseline by an average of 5.8\% in citation quality and 17.4\% in response correctness. The implementation is publicly available at https://github.com/BAI-LAB/c2cite

Figures (6)

• Figure 1: Illustration of integrating contextual citation (i.e., ours) v.s. using only citation markers (i.e., [i]) in response generation of LLMs. Without contextual information, LLMs may produce erroneous or fictitious citations.
• Figure 2: The architecture overview of C$^2$-Cite with two key components: Contextual-Aware Embedding and Contextual Citation Alignment components to transform passive placeholders into active knowledge pointers for citation symbols.
• Figure 3: Efficiency comparison via Throughput (samples/second) on three datasets.
• Figure 4: A case study of attention pattens in Naive LLM vs. C$^2$-Cite++ in generated responses.
• Figure 5: Degradation of response correctness with citation-integrated training data. Our model enhances response quality by incorporating contextual information.