xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

TL;DR

xRAG presents a novel modality-fusion approach to context compression in retrieval-augmented generation by projecting dense document embeddings into the language model space with a trainable modality bridge. By freezing the retriever and LM and inserting only a single document token, xRAG achieves extreme compression while maintaining or improving performance on knowledge-intensive tasks and reducing FLOPs by over 3×. The training pipeline combines paraphrase pretraining and context-aware instruction tuning (with language modeling and self-distillation objectives) to align the retrieved feature with LM representations. Across 7B and Mixtral-8x7B backbones, xRAG delivers robust improvements (average >10%) and substantial efficiency gains, demonstrating strong plug-and-play properties and suggesting a promising future direction for scalable retrieval-augmented systems.

Abstract

This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation. xRAG reinterprets document embeddings in dense retrieval--traditionally used solely for retrieval--as features from the retrieval modality. By employing a modality fusion methodology, xRAG seamlessly integrates these embeddings into the language model representation space, effectively eliminating the need for their textual counterparts and achieving an extreme compression rate. In xRAG, the only trainable component is the modality bridge, while both the retriever and the language model remain frozen. This design choice allows for the reuse of offline-constructed document embeddings and preserves the plug-and-play nature of retrieval augmentation. Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks, adaptable to various language model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several datasets, while reducing overall FLOPs by a factor of 3.53. Our work pioneers new directions in retrieval-augmented generation from the perspective of multimodality fusion, and we hope it lays the foundation for future efficient and scalable retrieval-augmented systems

Figures (10)

• Figure 1: xRAG enables efficient retrieval augmentation by adding one document token [X].
• Figure 2: Overview of xRAG (a) and RAG (b). For a given query, RAG typically concatenates the retrieved document with the query, significantly extending the context length. In contrast, xRAG addresses this issue through modality fusion by directly projecting the document embedding into the LLM's representation space. This allows for efficient retrieval-augmentation with the addition of only one token.
• Figure 3: Two-stage training strategy of xRAG including (a) Paraphrase Pre-training on unlabeled corpus and (b) Context-aware Instruction Tuning optimized with labeled data and self-distillation.
• Figure 4: Resilience rate and boost rate of three augmentation methods: LLMLinuga, xRAG and RAG over a Mixtral-8x7b baseline without retrieval augmentation.
• Figure 5: Given the misleading document, RAG model tend to generate a wrong answer based on the document, while xRAG demonstrate its robustness by leveraging the internal knowledge of LLM.