EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models

TL;DR

EfficientLLaVA tackles the deployment bottlenecks of multimodal large language models by automatically pruning LVLMs with only a few proxy samples. It integrates structural risk minimization to maximize pruning-policy generalization and employs search-space evolution by optimizing the vision projector, while using OBS to prune weight masks in LLaMA-based layers. The approach yields a favorable accuracy-efficiency trade-off, achieving $83.05\%$ on ScienceQA with a $1.8\times$ speedup and strong generalization across VizWiz, MMVet, and LLaVA-Bench, outperforming prior pruning methods at similar budgets. These results demonstrate practical, edge-friendly deployment of LVLMs with limited data without substantial sacrifices in multimodal reasoning capabilities.

Abstract

While multimodal large language models demonstrate strong performance in complex reasoning tasks, they pose significant challenges related to model complexity during deployment, especially for resource-limited devices. In this paper, we propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning. Conventional methods rely on the training data of the original model to select the proper pruning ratio for different network components. However, these methods are impractical for large vision-language models due to the unaffordable search costs caused by web-scale training corpus. In contrast, our approach only leverages a small number of samples to search for the desired pruning policy by maximizing its generalization ability on unknown training data while maintaining the model accuracy, which enables the achievement of an optimal trade-off between accuracy and efficiency for large visual language models. Specifically, we formulate the generalization gap of the pruning strategy using the structural risk minimization principle. Based on both task performance and generalization capability, we iteratively search for the optimal pruning policy within a given search space and optimize the vision projector to evolve the search space with higher upper bound of performance. We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering. Using only 64 samples for pruning policy search, EfficientLLaVA achieves an accuracy of 83.05% on ScienceQA, along with a $\times$ 1.8 speedup compared to the dense LLaVA-v1.5-7B model.

Figures (4)

• Figure 1: The trade-off between accuracy and inference speed on ScienceQA with EfficientLLaVA.
• Figure 2: The overall pipeline of EfficientLLaVA. In each iteration, we first search for the optimal pruning policy for matrix in each LLaMA layer, where evolutionary algorithms are employed with the fitness function containing model accuracy and generalization ability. Then we evolve the search space by optimizing the projector weight so that the upper bound of accuracy and generalization ability for all policies can be improved.
• Figure 3: The performance variation with the number of sampled data $n$ and sequence length $s$ in pruning space evolution tested on ScienceQA.
• Figure 4: Visual examples from LLaVA-SQA-7B. We color the text to show the response of different pruning methods and EfficientLLaVA consistently delivers more refined, contextually appropriate responses, showcasing its superior pruning and reasoning capabilities.