Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models

Xiaoran Fan, Zhichao Sun, Tao Ji, Lixing Shen, Tao Gui

TL;DR

This work tackles the KV cache explosion in vision-language models by adapting MHA/GQA-based VLMs to DeepSeek's MLA through a parameter-efficient framework. It introduces modality-adaptive partial-RoPE (MKL) and modality-decoupled SVD (MD-SVD) to compress KV space without extensive pretraining, aided by PEFT and activation-error minimization. Empirical results across LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL show substantial KV reduction with minimal performance loss, and strong compatibility with KV quantization and cache pruning baselines. The approach offers a practical, data-efficient path to scalable multimodal inference with MLA, advancing efficient cross-modal reasoning in real-world deployments.

Abstract

As vision-language models (VLMs) tackle increasingly complex and multimodal tasks, the rapid growth of Key-Value (KV) cache imposes significant memory and computational bottlenecks during inference. While Multi-Head Latent Attention (MLA) offers an effective means to compress the KV cache and accelerate inference, adapting existing VLMs to the MLA architecture without costly pretraining remains largely unexplored. In this work, we present MHA2MLA-VLM, a parameter-efficient and multimodal-aware framework for converting off-the-shelf VLMs to MLA. Our approach features two core techniques: (1) a modality-adaptive partial-RoPE strategy that supports both traditional and multimodal settings by selectively masking nonessential dimensions, and (2) a modality-decoupled low-rank approximation method that independently compresses the visual and textual KV spaces. Furthermore, we introduce parameter-efficient fine-tuning to minimize adaptation cost and demonstrate that minimizing output activation error, rather than parameter distance, substantially reduces performance loss. Extensive experiments on three representative VLMs show that MHA2MLA-VLM restores original model performance with minimal supervised data, significantly reduces KV cache footprint, and integrates seamlessly with KV quantization.

MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models

TL;DR

This work tackles the KV cache explosion in vision-language models by adapting MHA/GQA-based VLMs to DeepSeek's MLA through a parameter-efficient framework. It introduces modality-adaptive partial-RoPE (MKL) and modality-decoupled SVD (MD-SVD) to compress KV space without extensive pretraining, aided by PEFT and activation-error minimization. Empirical results across LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL show substantial KV reduction with minimal performance loss, and strong compatibility with KV quantization and cache pruning baselines. The approach offers a practical, data-efficient path to scalable multimodal inference with MLA, advancing efficient cross-modal reasoning in real-world deployments.

Abstract

As vision-language models (VLMs) tackle increasingly complex and multimodal tasks, the rapid growth of Key-Value (KV) cache imposes significant memory and computational bottlenecks during inference. While Multi-Head Latent Attention (MLA) offers an effective means to compress the KV cache and accelerate inference, adapting existing VLMs to the MLA architecture without costly pretraining remains largely unexplored. In this work, we present MHA2MLA-VLM, a parameter-efficient and multimodal-aware framework for converting off-the-shelf VLMs to MLA. Our approach features two core techniques: (1) a modality-adaptive partial-RoPE strategy that supports both traditional and multimodal settings by selectively masking nonessential dimensions, and (2) a modality-decoupled low-rank approximation method that independently compresses the visual and textual KV spaces. Furthermore, we introduce parameter-efficient fine-tuning to minimize adaptation cost and demonstrate that minimizing output activation error, rather than parameter distance, substantially reduces performance loss. Extensive experiments on three representative VLMs show that MHA2MLA-VLM restores original model performance with minimal supervised data, significantly reduces KV cache footprint, and integrates seamlessly with KV quantization.
Paper Structure (35 sections, 1 theorem, 15 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 35 sections, 1 theorem, 15 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. MHA2MLA-VLM
  3. Multimodal Partial-RoPE
  4. Full Vanilla RoPE
  5. Full Multimodal RoPE
  6. Multimodal Adaptive Partial-RoPE Strategies
  7. Modality-Decoupled SVD (MD-SVD)
  8. Unimodal SVD Baselines
  9. Modality-Decoupled SVD
  10. Motivation
  11. Experiment
  12. Setups
  13. Main Results
  14. MHA2MLA, Cache Pruning and Compression
  15. Ablation Study
  16. ...and 20 more sections

Key Result

Theorem 2.1

For a multimodal token sequence, we denote the joint activation matrix as $\bm{X}_{joint}\in\mathbb{R}^{d\times (n_v+n_t)}$ which can be partitioned into two modality-specific components: where $\bm{X}_{visual}\in\mathbb{R}^{d\times n_v}$ and $\bm{X}_{text}\in\mathbb{R}^{d\times n_t}$ denote the activations corresponding to visual and text tokens, respectively. Then the minimum loss of joint-moda

Figures (5)

  • Figure 1: The overview process of converting VLMs from MHA/GQA to MLA using MHA2MLA-VLM. Our method makes the attention inputs match MLA exactly, and low rank compression of the KV cache is consistent with MLA. The modality-decoupled design reduces truncation loss and maximizes the parameter reuse of pretrained weights.
  • Figure 2: Training loss curves of MHA2MLA (LLaVA-1.5) and GQA2MLA (LLaVA-NeXT) with different $d_{kv}$ settings.
  • Figure 3: Training loss of GQA2MLA on LLaVA-NeXT w and w/o MD-SVD initialization under different $d_{kv}$.
  • Figure 4: Quantitative analysis of MD-SVD via layer-wise loss ratio. Modality decoupled shows consistent improvements over joint optimization across all models.
  • Figure 5: Comparison of multimodal partial-rope selection between $\mathcal{S}_{\text{2-norm}}$ and $\mathcal{S}_{\text{MKL}}$.

Theorems & Definitions (2)

  • Theorem 2.1
  • proof