Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching

Hong Chen, Xiang Liu, Bo Wang, Yuxuan Fan, Yuanlin Chu, Zongluo Li, Xiaowen Chu, Xuming Hu

TL;DR

SONIC tackles the memory bottleneck of the Key-Value cache in multi-turn LLM inference by compressing historical turns into learnable Nexus tokens. It employs a turn-aware Nexus insertion, a hierarchical visibility mask, and an adaptive budget training regime, supported by multi-level distillation and reconstruction objectives to preserve semantic fidelity. The approach yields strong gains on MTBench101 and long-context benchmarks, with significant reductions in inference time and memory usage across 0.6B–4B Qwen models. This framework enables flexible, efficient deployment of long-context dialogue systems without retraining, representing a practical path toward scalable multi-turn LLM applications.

Abstract

The linear growth of Key-Value (KV) cache remains a bottleneck for multi-turn LLM deployment. Existing KV cache compression methods often fail to account for the structural properties of multi-turn dialogues, relying on heuristic eviction that risks losing critical context. We propose \textbf{SONIC}, a learning-based framework that compresses historical segments into compact and semantically rich \textbf{Nexus} tokens. By integrating dynamic budget training, SONIC allows flexible adaptation to varying memory constraints without retraining. Experiments show that at compression ratios of 80\% and 50\%, SONIC consistently outperforms baselines such as H2O and StreamingLLM on four diverse multi-turn benchmarks. Specifically, on the widely used MTBench101 benchmark, SONIC achieves an average score improvement of 35.55\% over state-of-the-art baselines, validating its effectiveness in sustaining coherent multi-turn dialogues. Furthermore, SONIC enhances deployment efficiency, accelerating the overall inference process by 50.1\% compared to full-context generation.

SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching

TL;DR

SONIC tackles the memory bottleneck of the Key-Value cache in multi-turn LLM inference by compressing historical turns into learnable Nexus tokens. It employs a turn-aware Nexus insertion, a hierarchical visibility mask, and an adaptive budget training regime, supported by multi-level distillation and reconstruction objectives to preserve semantic fidelity. The approach yields strong gains on MTBench101 and long-context benchmarks, with significant reductions in inference time and memory usage across 0.6B–4B Qwen models. This framework enables flexible, efficient deployment of long-context dialogue systems without retraining, representing a practical path toward scalable multi-turn LLM applications.

Abstract

The linear growth of Key-Value (KV) cache remains a bottleneck for multi-turn LLM deployment. Existing KV cache compression methods often fail to account for the structural properties of multi-turn dialogues, relying on heuristic eviction that risks losing critical context. We propose \textbf{SONIC}, a learning-based framework that compresses historical segments into compact and semantically rich \textbf{Nexus} tokens. By integrating dynamic budget training, SONIC allows flexible adaptation to varying memory constraints without retraining. Experiments show that at compression ratios of 80\% and 50\%, SONIC consistently outperforms baselines such as H2O and StreamingLLM on four diverse multi-turn benchmarks. Specifically, on the widely used MTBench101 benchmark, SONIC achieves an average score improvement of 35.55\% over state-of-the-art baselines, validating its effectiveness in sustaining coherent multi-turn dialogues. Furthermore, SONIC enhances deployment efficiency, accelerating the overall inference process by 50.1\% compared to full-context generation.
Paper Structure (32 sections, 15 equations, 4 figures, 6 tables)

This paper contains 32 sections, 15 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SONIC Framework
  4. Nexus Insertion and Representation
  5. Hierarchical Visibility Mask
  6. Training Strategy
  7. Multi-Level Knowledge Distillation
  8. Information Bottleneck and Reconstruction
  9. Adaptive Budget Training
  10. Benchmarks
  11. Experiments
  12. Settings
  13. Main Results
  14. Ablation Study
  15. Impact of Training Objectives
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Attention Pattern Visualization of SONIC model on MTBench101 Sample. U$X$ and A$X$ indicate the $X$-th turn of user and assistant messages, respectively. The original Tokens of historical rounds have been discarded and compressed into Nexus Tokens.
  • Figure 2: Overview of the SONIC framework. Historical segments are compressed into Nexus tokens, which are fully connected across turns. The hierarchical visibility mask ensures that only the system prompt, Nexus tokens, and the current query are accessible during generation.
  • Figure 3: Comparison of Low-score Proportions (scores 1–3) for Turns $\ge$ 4 on MTBench101. Results are shown for Qwen3-0.6B and Qwen3-4B models at an 80% compression rate. Lower proportions indicate superior multi-turn stability
  • Figure 4: Ablation Study on Impact of Adaptive Budget Training on model robustness. We evaluate the Qwen3-0.6B model on the CoreRes benchmark under an 80% compression ratio.