Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, Chao Huang

TL;DR

This work has identified a key pattern: certain seemingly meaningless separator tokens contribute disproportionately to attention scores compared to semantically meaningful tokens, which suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss.

Abstract

Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

TL;DR

This work has identified a key pattern: certain seemingly meaningless separator tokens contribute disproportionately to attention scores compared to semantically meaningful tokens, which suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss.

Abstract

Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.

Paper Structure

This paper contains 53 sections, 7 theorems, 18 equations, 11 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Fundamental Design
  5. Overall Pipeline
  6. Training/Pre-filling.
  7. Generation.
  8. Tailored Streaming Design
  9. Framework.
  10. Positional Encoding.
  11. Experiments and Results
  12. Experimental Settings
  13. Model.
  14. Training Datasets.
  15. Parameter Setting.
  16. ...and 38 more sections

Key Result

Theorem 5.1

Given $p>1$ and $n>2$, for any $\epsilon>0$ and $f \in \mathcal{F}$, there exists a SepLLM $g \in \mathcal{T}_{\text{Sep}}^{2,1,4}$, such that $d_{p}\left(f,g \right) < \epsilon$.

Figures (11)

  • Figure 1: The loss comparison between vanilla Transformer and the proposed SepLLM. SepLLM achieves lower loss w.r.t different computation costs and different training time consistently.
  • Figure 2: The visualization for attention scores of different layers given the input "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. ...". Note that the separator tokens like "," and "." contribute massive attentions.
  • Figure 3: The overall paradigm of SepLLM . The left side illustrates the attention mask in the training or pre-filling stage given the input "ABC$,$DE$.$FG$\backslash n$". The right side illustrates the KV cache management in the generation stage.
  • Figure 4: Overall framework of the proposed SepLLM tailored for streaming applications. The KV pairs are storaged in four cache blocks (displayed as four columns), and are updated in each iteration (shown in a single row). Once the runtime usage $Size_{run}$ reach the max capacity $\textbf{c}$, SepLLM move KV caches of separator tokens in Past Window Cache into Separator Cache and drop other KV caches.
  • Figure 5: Training loss curves for training from scratch. \ref{['fig:training_loss_b']} shows the ratios of the loss values of different methods to that of Vanilla with respect to FLOPs.
  • ...and 6 more figures

Theorems & Definitions (9)

  • Theorem 5.1
  • Theorem 10.1
  • Lemma 11.1: Lemma 5 in yun2020n
  • Lemma 11.2: Lemma 6 in yun2020n
  • Definition 11.3: Contextual Mapping
  • Lemma 11.4
  • proof
  • Lemma 11.5: Lemma 8 in yun2020n
  • Lemma 11.6: Lemma 4 in yun2020n