TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models

TL;DR

This work tackles the challenge of pruning Multimodal Large Language Models (MLLMs) without sacrificing multimodal capabilities. It introduces Token-Adaptive Multimodal Pruning (TAMP), which combines Diversity-Aware Sparsity (DAS) and Adaptive Multimodal Input Activation (AMIA) to tailor layer-wise sparsity and input-token selection to per-layer multimodal token distributions and attention patterns. Empirical results on LLaVA-NeXT and VideoLLaMA2 show that TAMP outperforms state-of-the-art pruning baselines across a wide range of benchmarks, particularly at high sparsity, and maintains diverse multimodal understanding. The method offers a resource-efficient path to deploy large, multimodal models in constrained environments by leveraging intrinsic token attributes rather than gradient/Hessian computations.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased model scale. While post-training pruning reduces model size in unimodal models, its application to MLLMs often yields limited success. Our analysis discovers that conventional methods fail to account for the unique token attributes across layers and modalities inherent to MLLMs. Inspired by this observation, we propose TAMP, a simple yet effective pruning framework tailored for MLLMs, featuring two key components: (1) Diversity-Aware Sparsity, which adjusts sparsity ratio per layer based on diversities among multimodal output tokens, preserving more parameters in high-diversity layers; and (2) Adaptive Multimodal Input Activation, which identifies representative multimodal input tokens using attention scores to guide unstructured weight pruning. We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities. Empirical experiments across various multimodal evaluation benchmarks demonstrate that each component of our approach substantially outperforms existing pruning techniques.

Figures (9)

• Figure 1: Illustration of multimodal token attributes. (Top): t-SNE visualization of multimodal output tokens of the layer, exhibiting unique distributions of each modality. (Bottom): Cosine similarity between the $\ell_{2}$-norm of tokens from each modality and all tokens, demonstrating a bias in input activations toward the modality with the largest token count ($N_a,N_v\!>>\!N_l$), resulting in suboptimal weight pruning.
• Figure 2: Overview of TAMP. Our method utilizes multimodal token attributes to guide MLLM pruning. (Left): To effectively preserve each MLLM layer's differing capability to encode rich multimodal output tokens after pruning, we apply layer-wise sparsity, assigning sparsity inversely to the layer's importance, which is computed as the average of intra-modality ($\mathbf{s}_{v}, \mathbf{s}_{l}$) and inter-modality ($\mathbf{s}_{vl}$) diversities (\ref{['sec:sub:diversity-aware-sparsity']}). (Right): To capture unique multimodal processing demands across different layers, we leverage attention scores to adaptively select multimodal input tokens for input activation calculations (\ref{['sec:sub:adaptive-selection']}).
• Figure 3: Intra-modality diversities ($\mathbf{s}_v, \mathbf{s}_l$) measure the average cosine distances among output tokens within the same modality, and inter-modality diversity ($\mathbf{s}_{vl}$) measures distances between output tokens from different modalities. We compute these diversities for each projection type in multi-head attention (Top) and feed-forward network (Bottom) across LLaVA-NeXT blocks using 128 randomly sampled inputs from the calibration set (ShareGPT4V). Notably, diversity trends differ by (1) modalities, (2) projection types, and (3) blocks, demonstrating varying capacities that should be preserved to effectively encode multimodal information across layers.
• Figure 4: Average attention score across LLaVA-NexT blocks. Varying attention scores indicate that unique multimodal processing demands exist for each block.
• Figure 5: Average relative performances of all pruning techniques at different sparsity ratios for the LLaVA-NeXT.