EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation

TL;DR

EXIT tackles the efficiency–effectiveness bottleneck in retrieval-augmented QA by introducing a context-aware, extractive compression framework that operates at the sentence level. It uses full-document context to perform parallelized Yes/No classification on sentences and reconstructs a coherent, ordered D' with a threshold $\tau$ to balance evidence retention and token reduction, significantly reducing end-to-end latency. Across single-hop and multi-hop QA benchmarks, EXIT outperforms abstractive and extractive baselines as well as uncompressed retrieval, achieving notable QA gains while cutting tokens by roughly 30% and reducing latency to around 0.8–3.5 seconds depending on the reader size. The approach is plug-and-play, model- and retriever-agnostic, and demonstrates robust generalization, including to domain-specific datasets and zero-shot supervision signals, highlighting its practical impact for scalable, high-quality RAG systems.

Abstract

We introduce EXIT, an extractive context compression framework that enhances both the effectiveness and efficiency of retrieval-augmented generation (RAG) in question answering (QA). Current RAG systems often struggle when retrieval models fail to rank the most relevant documents, leading to the inclusion of more context at the expense of latency and accuracy. While abstractive compression methods can drastically reduce token counts, their token-by-token generation process significantly increases end-to-end latency. Conversely, existing extractive methods reduce latency but rely on independent, non-adaptive sentence selection, failing to fully utilize contextual information. EXIT addresses these limitations by classifying sentences from retrieved documents - while preserving their contextual dependencies - enabling parallelizable, context-aware extraction that adapts to query complexity and retrieval quality. Our evaluations on both single-hop and multi-hop QA tasks show that EXIT consistently surpasses existing compression methods and even uncompressed baselines in QA accuracy, while also delivering substantial reductions in inference time and token count. By improving both effectiveness and efficiency, EXIT provides a promising direction for developing scalable, high-quality QA solutions in RAG pipelines. Our code is available at https://github.com/ThisIsHwang/EXIT

Figures (10)

• Figure 1: Average QA accuracy (EM) and efficiency (Total Latency) across various compression methods using Contriever-MSMARCO as the retriever and Llama-3.1-8b-Instruct as the reader. Experiments were conducted on a single A100-80GB GPU, and latency (in seconds) was averaged over all samples from these datasets.
• Figure 2: Overview of our framework. First, the retrieved document is split into sentences. Next, each sentence is classified as either "Yes" or "No" using the Compressor. Finally, sentences with scores above the threshold are recombined in their original order to complete the compression.
• Figure 3: Performance analysis on HQA across different Top-$k$ values (1, 5, 10, 20, 30), comparing accuracy, token retention, and inference latency between baselines and our method. All experiments were conducted on a single A100-80GB GPU.
• Figure 4: Comparison of compression and reading latency across baselines and our method in QA setting. Experiments were conducted on a single A100 GPU.
• Figure 5: Ablation on HQA comparing EM scores and latency for different model configurations within EXIT (red dot). CompAct and Original Docs are included as indicators. Experiments on a single A100-80GB GPU.