SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models

TL;DR

SmartTrim tackles the inefficiency of vision-language transformers by enabling per-input adaptive pruning of tokens and attention heads. It introduces two lightweight trimmers—XModal-Aware Token Trimmer and Modal-adaptive Head Trimmer—that are guided by cross-modal information and integrated into the backbone, complemented by a self-distillation training objective and a curriculum-based optimization schedule. The method achieves 2–4x inference speedups across multiple VL tasks with only 1–3% performance loss, outperforming static and prior adaptive acceleration approaches and can synergize with static compression techniques. This work significantly advances practical deployment of VL models by delivering fine-grained, input-aware efficiency without heavy pretraining or retraining requirements, enabling scalable, real-world use of large VL architectures.

Abstract

Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim.

Figures (12)

• Figure 1: FLOPs histogram of SmartTrim on VQA. SmartTrim allocates diverse computational overhead based on cross-modal complexity, assigning fewer computations to easy instances (left) and more to hard ones (right).
• Figure 2: Overview of our SmartTrim framework, best viewed in color. (a) Model Architecture of SmartTrim. We incorporate the trimmers into layers of the uni-modal encoders and the cross-modal encoder to prune redundant tokens and heads. Given a set of image-text pairs, SmartTrim adjusts the computations for each instance based on the trimmer outputs. (b) Self-Distillation strategy. At each training step, the predictions of the pruned model are aligned to its fully-capacity counterpart.
• Figure 3: The similarities in representations of tokens (top) and heads (bottom) in cross-modal encoder of METER fine-tuned on VQA.
• Figure 4: Pareto front of the efficiency-performance trade-offs of acceleration methods based on METER or BLIP backbones.
• Figure 5: Averaged latency on the VQA dataset.