ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models

TL;DR

This work tackles the high computational cost of pruning large vision-language models by introducing Efficient Coarse-to-Fine Layer-Wise Pruning (ECoFLaP). The method first estimates a global weight-importance score without Hessian computations using a zeroth-order gradient via forward-forward, to determine adaptive sparsity per layer, then applies layer-wise pruning with locally informed criteria (Wanda) within those budgets. ECoFLaP demonstrates superior performance over global pruning, SparseGPT, Wanda, and UPop across diverse VL and unimodal tasks, while substantially reducing GPU memory usage (up to ~40%). The approach generalizes to unimodal models and supports practical deployment of compact, high-performing VLLMs, with extensive ablations and visualizations validating the design choices.

Abstract

Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities, achieving remarkable advancements on various multimodal downstream tasks. However, deploying LVLMs is often problematic due to their massive computational/energy costs and carbon consumption. Such issues make it infeasible to adopt conventional iterative global pruning, which is costly due to computing the Hessian matrix of the entire large model for sparsification. Alternatively, several studies have recently proposed layer-wise pruning approaches to avoid the expensive computation of global pruning and efficiently compress model weights according to their importance within a layer. However, they often suffer from suboptimal model compression due to their lack of a global perspective. To address this limitation in recent efficient pruning methods for large models, we propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs. We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients. Then, the model performs local layer-wise unstructured weight pruning based on globally-informed sparsity ratios. We validate our proposed method across various multimodal and unimodal models and datasets, demonstrating significant performance improvements over prevalent pruning techniques in the high-sparsity regime.

Figures (6)

• Figure 1: (a) and (b): The imbalance of the magnitude and gradient distributions between vision and language models. (c): The skewed distribution of the layer-wise scores of SparseGPT.
• Figure 2: Illustration of our ECoFLaP compared to global and layer-wise pruning. The boxes with a blue, green, and black border denote the language, vision, and Q-Former modules, respectively. The dotted red arrows show the working flow of the algorithm. The beige color indicates the pruning of the current step (layer) is conditioned on the pruning decisions made in the preceding steps (layers), which are marked in gray. ECoFLaP first performs the efficient coarse step to obtain the pruning ratio for each layer by leveraging the zeroth-order gradient, and then removes the uncritical weights in a layer-wise fashion in the fine step.
• Figure 3: Illustration of (a) the zeroth-order gradient in weight space, and (b) the forward-forward algorithm.
• Figure 4: The accuracy and model sparsity trade-off for Wanda, ECoFLaP with first-order and zeroth-order gradient on both BLIP-2 (multimodal), FlanT5 (unimodal), and EVA-ViT (unimodal).
• Figure 5: The loss landscape of the BLIP-2 model.