Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

Chensen Huang, Guibo Zhu, Xuepeng Wang, Yifei Luo, Guojing Ge, Haoran Chen, Dong Yi, Jinqiao Wang

TL;DR

This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space, and investigates the issue of poor model responses when both instructions and context are compressed in downstream tasks.

Abstract

To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

TL;DR

This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space, and investigates the issue of poor model responses when both instructions and context are compressed in downstream tasks.

Abstract

To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer
Paper Structure (24 sections, 1 equation, 4 figures, 4 tables)

This paper contains 24 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Design of RCC
  4. Method Overview
  5. RCC Model Architecture
  6. RCC Encoder
  7. RCC Decoder
  8. Model Training Tasks
  9. Long Text Training Methods
  10. Instruction Reconstruction Method
  11. Experiments
  12. Text Reconstruction
  13. Passkey Retrieval Task
  14. Long-Text Benchmark Evaluation
  15. Evaluation Dataset
  16. ...and 9 more sections

Figures (4)

  • Figure 1: GPU memory Consumption of Different Models with Increasing Length. Left: Pythia-1.4b, Right: RCC model using Pythia-1.4b for both encoder and decoder. Both models utilize FlashAttention-2 dao2023flashattention2. A more detailed analysis of GPU memory consumption can be found in Appendix \ref{['ap5']}.
  • Figure 2: The structure of the encoder and decoder in RCC layer i. The maximum context window of the encoder is 2048. The encoder has a compression rate of 32, and we use the vectors at positions that are multiples of 32 in the output as the compressed vectors. Each segment will generate a compressed vector of length 64. When the sequence length exceeds 2048, the encoder performs cyclic segmentation and compression. The compressed vectors produced between segments in the encoder are independent, while those generated within a segment are correlated. The decoder's input is the residual connection between the input vector from the previous layer and the compressed vector after linear mapping. All compressed vectors will interact within the decoder.
  • Figure 3: Text reconstruction score
  • Figure 4: When the GPU memory approaches 60GB, the memory occupation of different models. Left: Pythia-1.4b, Right: RCC model using Pythia-1.4b for both encoder and decoder. Both models utilize FlashAttention-2 dao2023flashattention2.