Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model

TL;DR

This paper argues that data distribution, not just model architecture, dictates the effectiveness of context compression for long-context LLMs. It introduces an autoencoder framework trained from scratch across multiple data distributions to separately probe input entropy and intrinsic data gaps between encoder and decoder. Key findings show encoder-derived entropy harms compression, decoder priors bias complexity assessments, and that alignment with the decoder’s intrinsic data distribution is the most impactful factor, often outweighing pure model scaling. The work provides practical guidelines—prioritize decoder-aligned data and allocate more compute to the decoder—to achieve better, more data-efficient compression in real-world long-context tasks. Overall, it shifts context compression research toward data distribution awareness, with implications for RAG, ICL, and large-scale code or math domains.

Abstract

The deployment of Large Language Models (LLMs) in long-context scenarios is hindered by computational inefficiency and significant information redundancy. Although recent advancements have widely adopted context compression to address these challenges, existing research only focus on model-side improvements, the impact of the data distribution itself on context compression remains largely unexplored. To bridge this gap, we are the first to adopt a data-centric perspective to systematically investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data (i.e., the model's internal pretrained knowledge). We evaluate the semantic integrity of compressed representations using an autoencoder-based framework to systematically investigate it. Our experimental results reveal that: (1) encoder-measured input entropy negatively correlates with compression quality, while decoder-measured entropy shows no significant relationship under a frozen-decoder setting; and (2) the gap between intrinsic data of the encoder and decoder significantly diminishes compression gains, which is hard to mitigate. Based on these findings, we further present practical guidelines to optimize compression gains.

Figures (6)

• Figure 1: A illustration of an encoder-decoder architecture for context compression. Input data such as web pages, text, or code is mapped by the encoder into a latent representation $p(z|x)$, where different types of input data exhibit varying entropy values. The compressed tokens are then fed through the decoder to produce an output distribution $p(y|z)$. The dashed arrow denotes the intrinsic data gap between the encoder and the decoder. Our goal is to investigate, from a data-centric perspective, how input data with various entropy values and the intrinsic data (i.e., the model’s internal pretrained knowledge) gap impact the compression quality.
• Figure 2: The overall framework. Our framework includes two phases: (a) pretraining and (b) fine-tuning. Pretraining Phase: We begin by independently pre-training a randomly initialized encoder and decoder. To establish an intrinsic data discrepancy between the two modules, the encoder is pre-trained on dataset $\mathcal{D}_1$, whereas the decoder is pre-trained on a disjoint set $\mathcal{D}_{i \neq 1}$. The optimization objective for this phase is the standard negative log-likelihood loss $\mathcal{L}_{\text{nll}}$. Fine-tuning Phase: Following pre-training, the encoder and decoder are coupled to form a joint autoencoder architecture, where $e(\cdot)$ denote the embedding look-up function that maps tokens to their latent representations. To guide the decoder, we introduce a specialized indicator token $\texttt{[AE]}$, which prompts the decoder to perform the context reconstruction task. . During this stage, the model is trained using the autoencoder reconstruction loss $\mathcal{L}_{\text{AE}}$. Crucially, we optimize the encoder parameters while keeping the decoder frozen.
• Figure 3: The impact of input data entropy on the compression process. For the encoder, input data entropy is negatively correlated with compression quality. In contrast, for the decoder, input data entropy shows no clear association with compression quality.The encoder and decoder sizes are fixed at 500M to focus on the impact of data distribution.
• Figure 4: The impact of intrinsic data (i.e., pretrained knowledge) gap on the compression process. The encoder's pretraining data always comes from dataset $\mathcal{D}_1$, while the decoder's pretraining data comes from dataset $\mathcal{D}_1$ to dataset $\mathcal{D}_6$, as indicated by the horizontal axis. The test data is fixed as $\mathcal{D}_1$. As the divergence between the encoder and decoder increases, all metrics exhibit a clear downward trend. The encoder and decoder sizes are fixed at 500M to focus on the impact of data distribution.
• Figure 5: FLOPs comparison across various encoder-decoder size combinations. The y-axis reports the F1 score, and the x-axis shows the average compute cost in FLOPs per generation (log scale). $E_{\mathcal{D}_i}$(500M)+$D_{\mathcal{D}_i}(*)|$$\mathcal{D}_i$ indicates that the encoder backbone is fixed to $LLM_{\mathcal{D}_i}$(500M) while the decoder size is varied (and vice versa). $|\mathcal{D}_i$ denotes the data distribution used in the unified fine-tuning framework and for evaluation. Because different distributions contain different numbers of tokens, the computed FLOPs will vary accordingly.