Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension

Jushi Kai, Boyi Zeng, Yixuan Wang, Haoli Bai, Ziwei He, Bo Jiang, Zhouhan Lin

TL;DR

The paper tackles the challenge of limited context windows and quadratic attention costs in decoder-only LLMs by introducing FreqKV, a frequency-domain KV compression technique. KV states are transformed to the frequency domain via DCT, where low-frequency components—carrying most information—are retained and high-frequency components discarded, enabling iterative, on-the-fly context window extension without architectural changes. With minimal fine-tuning, LLMs learn to leverage the compressed KV cache, achieving competitive long-context perplexities and state-of-the-art results on long-context understanding benchmarks while maintaining low decoding overhead. This method offers a practical pathway to deploy longer-context LLMs with reduced memory and computation, improving scalability and applicability in real-world tasks.

Abstract

Frequency-domain compression has proven effective in reducing redundancies for spatial signals. In this work, we propose FreqKV, a novel frequency domain key-value (KV) compression technique that enables efficient context window extension for decoder-only large language models (LLMs). Our approach is motivated by a key observation that, in the frequency domain, the energy distribution of the KV cache is predominantly concentrated in low-frequency components. By discarding high-frequency components, we achieve efficient compression of the KV cache with minimal information loss. FreqKV iteratively compresses the increasing KV cache to a fixed size in the frequency domain, allowing models to process lengthy contexts efficiently. Introducing no additional parameters or architectural modifications, FreqKV is applicable to both fine-tuning and inference. With minimal fine-tuning, LLMs can learn to leverage the limited cache that is compressed in the frequency domain and extend the context window. Experiments on a range of long context language modeling and understanding tasks demonstrate the efficiency and effectiveness of the proposed method.

FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension

TL;DR

The paper tackles the challenge of limited context windows and quadratic attention costs in decoder-only LLMs by introducing FreqKV, a frequency-domain KV compression technique. KV states are transformed to the frequency domain via DCT, where low-frequency components—carrying most information—are retained and high-frequency components discarded, enabling iterative, on-the-fly context window extension without architectural changes. With minimal fine-tuning, LLMs learn to leverage the compressed KV cache, achieving competitive long-context perplexities and state-of-the-art results on long-context understanding benchmarks while maintaining low decoding overhead. This method offers a practical pathway to deploy longer-context LLMs with reduced memory and computation, improving scalability and applicability in real-world tasks.

Abstract

Frequency-domain compression has proven effective in reducing redundancies for spatial signals. In this work, we propose FreqKV, a novel frequency domain key-value (KV) compression technique that enables efficient context window extension for decoder-only large language models (LLMs). Our approach is motivated by a key observation that, in the frequency domain, the energy distribution of the KV cache is predominantly concentrated in low-frequency components. By discarding high-frequency components, we achieve efficient compression of the KV cache with minimal information loss. FreqKV iteratively compresses the increasing KV cache to a fixed size in the frequency domain, allowing models to process lengthy contexts efficiently. Introducing no additional parameters or architectural modifications, FreqKV is applicable to both fine-tuning and inference. With minimal fine-tuning, LLMs can learn to leverage the limited cache that is compressed in the frequency domain and extend the context window. Experiments on a range of long context language modeling and understanding tasks demonstrate the efficiency and effectiveness of the proposed method.
Paper Structure (26 sections, 8 equations, 7 figures, 6 tables)

This paper contains 26 sections, 8 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Context Extension for LLMs.
  4. KV Compression for LLMs.
  5. Learning in the Frequency Domain.
  6. Preliminaries
  7. Discrete Cosine Transform
  8. Self-Attention
  9. FreqKV
  10. Energy Concentration in the Frequency Domain
  11. KV Compression in the Frequency Domain
  12. Context Extension via Iterative Compression
  13. Experiments
  14. Implementation
  15. Long Context Language Modeling
  16. ...and 11 more sections

Figures (7)

  • Figure 1: The average power spectrums of key states and value states in different layers of LLaMA-2-7b. 1000 documents are sampled from CNN/Daily Mail cnndm. We use DCT to transform key states and value states to the frequency domain, and average power spectrums over these samples and hidden dimensions.
  • Figure 2: The illustration of our FreqKV. The KV cache will be compressed in an iterative manner as the cache reaches the context window size. Sink tokens remain uncompressed throughout the process. The tokens after sink tokens will be compressed in the frequency domain and subsequent tokens will continue to get into the cache. When the cache is filled again, the compressed tokens and incoming tokens will be compressed together. The compression is performed iteratively to extend the context window.
  • Figure 3: The Needle-in-a-Haystack results, with the x-axis representing the document length ranging from 1K to 8K tokens for LLaMA-2-chat-7b and 1K to 16K tokens for LLaMA-3-instruct-8b, and the y-axis showing the position of the "needle" within the document.
  • Figure 4: Perplexity evaluation on the validation set of PG-19 with different retaining ratios.
  • Figure 5: Decoding time with the full cache and FreqKV on the increasing sequence length.
  • ...and 2 more figures