Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference

Kunxi Li, Yufan Xiong, Zhonghua Jiang, Yiyun Zhou, Zhaode Wang, Chengfei Lv, Shengyu Zhang

TL;DR

FlowMM tackles the memory and latency challenges of multimodal large language models by introducing a cross-modal information flow-guided KV cache merging framework that adapts merging strategies to layer-specific interaction patterns and preserves task-critical information via sensitivity-aware token matching. The method identifies cross-modal flow through layer-wise attention and employs a pivot-based merging scheme that selectively merges low-risk tokens while protecting high-sensitivity tokens, all without fine-tuning. Empirical results across diverse MLLMs and MileBench tasks show FlowMM achieves 80%–95% KV cache reduction and 1.3x–1.8x decoding speedups with competitive performance, outperforming both eviction and unimodal merging baselines, especially on long-context multimodal tasks. The approach holds practical significance for scalable multimodal inference and points to future extensions toward video-audio modalities with longer temporal and higher-dimensional features.

Abstract

Traditional KV cache eviction strategies, which discard less critical KV-pairs based on attention scores, often degrade generation quality, causing context loss or hallucinations. Recent efforts shift toward KV merging, merging eviction tokens with retention tokens based on similarity. However, in multimodal scenarios, distributional biases across modality tokens and attentional biases in cross-modal interactions limit its effectiveness. This work introduces FlowMM, an adaptive framework for cross-modal information flow-guided multimodal KV cache merging. FlowMM leverages cross-modal information flow to dynamically apply layer-specific merging strategies, capturing modality-specific patterns while preserving contextual integrity. Furthermore, we introduce a sensitivity-adaptive token matching mechanism that jointly evaluates token similarity and task-critical sensitivity, merging low-risk tokens while safeguarding high-sensitivity ones. Extensive experiments across diverse leading MLLMs show that FlowMM reduces KV cache memory by 80% to 95% and decoding latency by 1.3-1.8x, while maintaining competitive task performance.

FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference

TL;DR

FlowMM tackles the memory and latency challenges of multimodal large language models by introducing a cross-modal information flow-guided KV cache merging framework that adapts merging strategies to layer-specific interaction patterns and preserves task-critical information via sensitivity-aware token matching. The method identifies cross-modal flow through layer-wise attention and employs a pivot-based merging scheme that selectively merges low-risk tokens while protecting high-sensitivity tokens, all without fine-tuning. Empirical results across diverse MLLMs and MileBench tasks show FlowMM achieves 80%–95% KV cache reduction and 1.3x–1.8x decoding speedups with competitive performance, outperforming both eviction and unimodal merging baselines, especially on long-context multimodal tasks. The approach holds practical significance for scalable multimodal inference and points to future extensions toward video-audio modalities with longer temporal and higher-dimensional features.

Abstract

Traditional KV cache eviction strategies, which discard less critical KV-pairs based on attention scores, often degrade generation quality, causing context loss or hallucinations. Recent efforts shift toward KV merging, merging eviction tokens with retention tokens based on similarity. However, in multimodal scenarios, distributional biases across modality tokens and attentional biases in cross-modal interactions limit its effectiveness. This work introduces FlowMM, an adaptive framework for cross-modal information flow-guided multimodal KV cache merging. FlowMM leverages cross-modal information flow to dynamically apply layer-specific merging strategies, capturing modality-specific patterns while preserving contextual integrity. Furthermore, we introduce a sensitivity-adaptive token matching mechanism that jointly evaluates token similarity and task-critical sensitivity, merging low-risk tokens while safeguarding high-sensitivity ones. Extensive experiments across diverse leading MLLMs show that FlowMM reduces KV cache memory by 80% to 95% and decoding latency by 1.3-1.8x, while maintaining competitive task performance.

Paper Structure

This paper contains 25 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficient Inference for Large Language Models
  4. KV Cache Compression
  5. Method
  6. Preliminary
  7. Observation
  8. Cross-modal Information Patterns.
  9. Cross-Modal Information Flow Merging.
  10. FlowMM
  11. Information Flow Guided Merging.
  12. Sensitivity-Adaptive Token Matching.
  13. Experiments
  14. Experimental settings
  15. Main result
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Comparison between Eviction-Based Compression (a), Traditional Merging-Based Compression (b), and our Cross-Modal Information Flow Guided Merging (c).
  • Figure 2: Overview of FlowMM. (a) Cross-modal information flow analysis determines whether each layer exhibits predominantly cross-modal or intra-modal interactions, enabling layer-specific merging strategies. (b) Sensitivity-adaptive token matching jointly considers token similarity and sensitivity scores, preserving high-sensitivity tokens while merging similar low-sensitivity tokens to maintain critical contextual information.
  • Figure 3: (a) Layer-wise divergence in cross-modal attention. (b) The performance comparison between full cache and information-flow merging.
  • Figure 4: Evaluation results of FlowMM and other KV cache compression methods with varied cache budgets.