Simple Context Compression: Mean-Pooling and Multi-Ratio Training

TL;DR

This work investigates reducing the cost of reasoning over long contexts in retrieval-augmented generation by introducing a simple mean-pooling compressor that encodes a long document into $C$ vectors via non-overlapping pooling, with $C = \left\lceil L / r \right\rceil$. The compressor uses a transformer encoder with full self-attention and shares parameters with the downstream decoder, requiring no extra learned components beyond the encoder and decoder, and is trained via knowledge distillation from a full-context teacher. A key contribution is the unified multi-ratio training objective, enabling a single compressor to support multiple compression budgets with only modest performance loss, and a thorough comparison against compression-tokens (including bidirectional attention variants) across six QA datasets, three model families, and multiple scales. The results show that mean-pooling consistently outperforms compression-tokens, benefits from model scaling, and that multi-ratio training can substantially reduce the need for separate models per ratio, though the gains are nuanced and depend on ratio and architecture. The study emphasizes the need for standardized evaluation in soft context compression and demonstrates practical implications for efficient long-context reasoning in LLM-based systems.

Abstract

A common strategy to reduce the computational costs of using long contexts in retrieval-augmented generation (RAG) with large language models (LLMs) is soft context compression, where the input sequence is transformed into a shorter continuous representation. We develop a lightweight and simple mean-pooling approach that consistently outperforms the widely used compression-tokens architecture, and study training the same compressor to output multiple compression ratios. We conduct extensive experiments across in-domain and out-of-domain QA datasets, as well as across model families, scales, and compression ratios. Overall, our simple mean-pooling approach achieves the strongest performance, with a relatively small drop when training for multiple compression ratios. More broadly though, across architectures and training regimes the trade-offs are more nuanced, illustrating the complex landscape of compression methods.

Figures (3)

• Figure 1: Comparison of context processing methods: regular LM, compression tokens, and our proposed approach --- mean pooling. The figure illustrates a compression ratio of $4\times$.
• Figure 2: Compression Scaling. We show the teacher-normalized $F_1$ scores (Relative F1) across four Qwen3 model scales. The scores are averages of the scores of all datasets. We can clearly observe the benefits of scaling for LLM compressors.
• Figure 3: In-domain and out-of-domain comparison. (a) Line plots show performance on in-domain vs. out-of-domain datasets. (b) Bar plots show the in–out performance gap per method, which is the difference between in-domain and out-of-domain teacher-normalized $F_1$ scores.