OSCAR: Online Soft Compression And Reranking

TL;DR

This work tackles the computational bottleneck of Retrieval-Augmented Generation (RAG) as retrieval scales, introducing OSCAR, an online, query-dependent soft compression method with optional document reranking. OSCAR builds two compressor variants (OSCAR-N-Layers and OSCAR-llama) that convert retrieved documents into a compact, query-conditioned embedding representation, enabling faster generation with minimal accuracy loss across 1B–24B parameter backbones. The training framework relies on distillation from teacher LLMs, with an objective that combines generation loss and an optional reranking loss, and does not require ground-truth labels. Empirically, OSCAR achieves 2–5× faster inference, 50–75% memory savings, and competitive or superior accuracy compared to hard-pruning baselines (Provence, ReCOMP) and soft offline methods (PISCO), while enabling joint reranking in a RAG pipeline and showing robustness across backbones and retrieval settings. These results suggest that online soft compression can substantially improve the practicality and scalability of RAG systems in real-world deployments.

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge, leading to improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as retrieval sizes grow. To address this, we introduce OSCAR, a novel query-dependent online soft compression method that reduces computational overhead while preserving performance. Unlike traditional hard compression methods, which shorten retrieved texts, or soft compression approaches, which map documents to continuous embeddings offline, OSCAR dynamically compresses retrieved information at inference time, eliminating storage overhead and enabling higher compression rates. Additionally, we extend OSCAR to simultaneously perform reranking, further optimizing the efficiency of the RAG pipeline. Our experiments demonstrate state-of-the-art performance with a 2-5x speed-up in inference and minimal to no loss in accuracy for LLMs ranging from 1B to 24B parameters. The models are available at: https://huggingface.co/collections/naver/oscar-67d446a8e3a2551f57464295.

Figures (16)

• Figure 2: OSCAR models enable faster end-to-end inference with retrieval as well as improved accuracy compared to hard compression methods.
• Figure 3: Overview of OSCAR inference (left) and training (right).
• Figure 4: LLM evaluation scores of each Mistral-7B-backboned models, in relation with the total number of floating point operations required at inference. OSCAR models are faster and more effective on most datasets. OSCAR-llama in particular offers the best alternative.
• Figure 5: GPT-4 pairwise comparisons. OSCAR-llama, while faster, is on par -- or better -- than Provence, Recomp and PISCO.
• Figure 6: Testing the robustness of OSCAR to retrieval changes. OSCAR models performance drops are similar to the uncompressed backbone.