Context Compression via Explicit Information Transmission

TL;DR

This work proposes ComprExIT (Context Compression via Explicit Information Transmission), a lightweight framework that formulates soft compression into a new paradigm: explicit information transmission over frozen LLM hidden states, demonstrating that explicit and coordinated information transmission enables more effective and robust long-context compression.

Abstract

Long-context inference with Large Language Models (LLMs) is costly due to quadratic attention and growing key-value caches, motivating context compression. In this work, we study soft context compression, where a long context is condensed into a small set of continuous representations. Existing methods typically re-purpose the LLM itself as a trainable compressor, relying on layer-by-layer self-attention to iteratively aggregate information. We argue that this paradigm suffers from two structural limitations: (i) progressive representation overwriting across layers (ii) uncoordinated allocation of compression capacity across tokens. We propose ComprExIT (Context Compression via Explicit Information Transmission), a lightweight framework that formulates soft compression into a new paradigm: explicit information transmission over frozen LLM hidden states. This decouples compression from the model's internal self-attention dynamics. ComprExIT performs (i) depth-wise transmission to selectively transmit multi-layer information into token anchors, mitigating progressive overwriting, and (ii) width-wise transmission to aggregate anchors into a small number of slots via a globally optimized transmission plan, ensuring coordinated allocation of information. Across six question-answering benchmarks, ComprExIT consistently outperforms state-of-the-art context compression methods while introducing only ~1% additional parameters, demonstrating that explicit and coordinated information transmission enables more effective and robust long-context compression.

Figures (8)

• Figure 1: Visualization of two structural weaknesses of existing LLM-as-a-compressor methods. (i) Representation overwriting: The information carried by the compression token that captures Year 2012 is gradually overwritten into highly abstract features during the layer-by-layer encoding, leading to information loss for the decoder. (ii) Lack of global coordination: The key token Paris is not attended by the compression tokens due to the lack of global coordination of compression capacity allocation.
• Figure 2: A comparison between existing LLM-as-a-compressor methods (left) and ComprExIT (right). Existing methods introduce gist tokens that are iteratively encoded by the self-attention layers in the LLMs, which are trained to aggregate information from context tokens and align the representations to the decoder's input space. ComprExIT instead leverages the hidden states of the context tokens encoded in a forward pass. The hidden states across layers are selectively aggregated into token anchors, which are then transmitted to the compression tokens through a coordinated transmission plan.
• Figure 3: Pearson correlation between compression slots/tokens’ aggregation distributions. For each compression slot, we treat its normalized aggregation weights over input tokens (i.e., the attention weights in ICAE (left) or the transmission plan in ComprExIT (right)) as a vector, and compute pairwise Pearson correlation between these vectors across slots. Large off-diagonal values indicate that different slots aggregate highly overlapping subsets of tokens, reflecting duplicated allocation behavior. As shown, ICAE exhibits substantial off-diagonal correlations with large high correlation areas (marked out in the figure), whereas ComprExIT maintains low inter-slot correlation, indicating a more coordinated information allocation.
• Figure 4: Singular value spectrum (normalized) and effective rank (erank in the figure) of the aggregation matrix of compression tokens, where compression slot has a vector of aggregation weights over input tokens. The spectrum shows much lower effective rank of ICAE's aggregation matrix compared with ComprExIT, indicating much higher allocation redundancies in compression.
• Figure 5: Depth-wise gating weights across layers in ComprExIT. The weights represent ComprExIT's preference for layers at the position of each input token.