Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors

TL;DR

This paper tackles the cost of processing long contexts in LLMs by introducing Semantic-Anchor Compression (SAC), an autoencoding-free context compression method. SAC selects anchor tokens $S$ directly from the input context, augments them with dedicated anchor embeddings, and uses bidirectional attention to aggregate contextual information into their KV representations, yielding a compact $\tilde{M}$. By avoiding autoencoding pretraining, SAC aligns compression with downstream tasks and demonstrates strong MRQA performance across various compression ratios. Ablation and visualization analyses show SAC achieves more locality and semantic alignment than prior methods, enabling better generalization and efficiency in long-context inference.

Abstract

Context compression presents a promising approach for accelerating large language model (LLM) inference by compressing long contexts into compact representations. Current context compression methods predominantly rely on autoencoding tasks to train context-agnostic compression tokens to compress contextual semantics. While autoencoding tasks enable compression tokens to acquire compression capabilities, compression via autoencoding tasks creates a fundamental mismatch: the models are optimized for reconstruction that diverge from actual downstream tasks, thereby weakening the features more beneficial for real-world usage. We propose Semantic-Anchor Compression (SAC), a novel method that shifts from autoencoding task based compression to an architecture that is equipped with this compression capability \textit{a priori}. Instead of training models to compress contexts through autoencoding tasks, SAC directly selects so-called anchor tokens from the original context and aggregates contextual information into their key-value (KV) representations. By deriving representations directly from the contextual tokens, SAC eliminates the need for autoencoding training. To ensure compression performance while directly leveraging anchor tokens, SAC incorporates two key designs: (1) anchor embeddings that enable the compressor to identify critical tokens, and (2) bidirectional attention modification that allows anchor tokens to capture information from the entire context. Experimental results demonstrate that SAC consistently outperforms existing context compression methods across various compression ratios. On out-of-distribution evaluation using MRQA, SAC achieves 1 EM improvement at 5x compression over strong baselines, with increasing advantages at higher compression ratios.

Figures (9)

• Figure 1: Three tasks for training the context compressor introduced by ICAE and followed by numerous works. The training uses (a) Autoencoding task and (b) Language modeling task to pretrain the encoder, then finetunes on (c) Question answering task.
• Figure 2: The difference between SAC and previous methods. While previous methods (a) compress contextual information into context-agnostic special tokens (referred to as compression tokens), SAC (b) compresses the context directly into the original contextual tokens themselves. Here, $\tilde{M}$ can represent either the output from the final layer of the LLM or the Key-Value pairs, being later on used as compressed representations for LLM inference.
• Figure 3: Key differentiators within SAC model architecture. (a) Representative tokens are transformed into anchor tokens through anchor embeddings. (b) The encoder in SAC adopts bidirectional attention, while the decoder operates with causal attention.
• Figure 4: Attention maps of different models finetuned under a 5x compression rate. From top to bottom, the figure displays the final layer attention maps for the 500xCompressor and SAC models, respectively. The x-axis represents the original context tokens, and the y-axis represents the compression/anchor tokens.
• Figure 5: The t-SNE visualization shows the key representations of the final layer KV values for SAC, 500xCompressor li2024500xcompressorgeneralizedpromptcompression, and EPL zhao2025positionidsmatterenhanced, respectively.