Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens

Shaoshen Chen, Yangning Li, Zishan Xu, Yinghui Li, Xin Su, Zifei Shan, Hai-tao Zheng

TL;DR

DAST addresses the inefficiency of long-context processing by dynamically allocating soft tokens to context chunks using the LLM’s own relevance signals. By combining perplexity-based local information with attention-based global information, it adaptively concentrates compression capacity on information-dense regions without external guidance. Empirical results on LongBench and NaturalQuestions show that DAST outperforms state-of-the-art static and vector-based compression methods, especially in preserving critical content and improving downstream task performance. The approach has practical impact for enabling efficient, scalable long-context reasoning in LLMs, with robust performance across varying compression levels. $\alpha$-sensitivity is stable, and the method remains applicable without heavy parameter tuning.$

Abstract

Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods.

DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens

TL;DR

DAST addresses the inefficiency of long-context processing by dynamically allocating soft tokens to context chunks using the LLM’s own relevance signals. By combining perplexity-based local information with attention-based global information, it adaptively concentrates compression capacity on information-dense regions without external guidance. Empirical results on LongBench and NaturalQuestions show that DAST outperforms state-of-the-art static and vector-based compression methods, especially in preserving critical content and improving downstream task performance. The approach has practical impact for enabling efficient, scalable long-context reasoning in LLMs, with robust performance across varying compression levels. -sensitivity is stable, and the method remains applicable without heavy parameter tuning.$

Abstract

Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods.

Paper Structure

This paper contains 16 sections, 5 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Compression Background
  4. Overall Framework
  5. Dynamic Allocation
  6. Experiments
  7. Main Results
  8. Comparison of Different Constraints
  9. Ablation Study
  10. Parameter Sensitivity Analysis
  11. Conclusion
  12. Limitations
  13. Reallocation Algorithm
  14. Settings
  15. Implementation
  16. ...and 1 more sections

Figures (3)

  • Figure 1: (a) and (b): the previous fixed allocation methods include Single-chunk and Multi-chunks compression; (c): our dynamic allocation method. Notably, our dynamic method allocated more soft tokens to the key information in the answer (highlighted in bold) while reducing soft tokens for less information. $\text{<CT>}$ is compress soft token.
  • Figure 2: Performance and Number of soft tokens v.s. Key Information Position.
  • Figure 3: Parameter Sensitivity Analysis of $\alpha$.