Straightforward Layer-wise Pruning for More Efficient Visual Adaptation

TL;DR

The paper addresses inefficiencies in parameter-efficient transfer learning (PETL) caused by frozen pre-trained weights that leave redundant structures. It proposes Straightforward Layer-wise Pruning (SLS), a training-free method that prunes entire layers based on a feature-perspective using $t$-SNE and the Silhouette Coefficient Index (SC\_Index), with a threshold $T = \alpha \times SC\_Index_N$ to determine pruning depth. SLS preserves PETL’s storage footprint while enabling improved throughput, demonstrated across ViT and Swin backbones on VTAB-1k, as well as object detection and segmentation tasks, outperforming mainstream pruning methods like DepGraph. The approach reduces storage overhead by storing only pruning indices, scales across diverse downstream datasets, and maintains accuracy with minimal degradation, highlighting its practical impact for efficient deployment of large pretrained vision models.

Abstract

Parameter-efficient transfer learning (PETL) aims to adapt large pre-trained models using limited parameters. While most PETL approaches update the added parameters and freeze pre-trained weights during training, the minimal impact of task-specific deep layers on cross-domain data poses a challenge as PETL cannot modify them, resulting in redundant model structures. Structural pruning effectively reduces model redundancy; however, common pruning methods often lead to an excessive increase in stored parameters due to varying pruning structures based on pruning rates and data. Recognizing the storage parameter volume issue, we propose a Straightforward layer-wise pruning method, called SLS, for pruning PETL-transferred models. By evaluating parameters from a feature perspective of each layer and utilizing clustering metrics to assess current parameters based on clustering phenomena in low-dimensional space obtained through t-SNE, SLS facilitates informed pruning decisions. Our study reveals that layer-wise pruning, with a focus on storing pruning indices, addresses storage volume concerns. Notably, mainstream Layer-wise pruning methods may not be suitable for assessing layer importance in PETL-transferred models, where the majority of parameters are pre-trained and have limited relevance to downstream datasets. Comparative analysis against state-of-the-art PETL methods demonstrates that the pruned model achieved a notable balance between model throughput and accuracy. Moreover, SLS effectively reduces storage overhead arising from varying pruned structures while enhancing the accuracy and speed of pruned models compared to conventional pruning methods.

Figures (6)

• Figure 1: Comparison of Parameter-efficient Transfer Learning, Mainstream Pruning, and our SLS. SLS can selectively prune task-specific layers that cannot be updated by PETL, without increasing the number of stored parameters.
• Figure 1: The pruning outcomes of Convpass on VTAB-1K and the corresponding precision of the model before and after pruning.
• Figure 2: The outcomes of decreasing the dimensionality of RepAdapterrepadapter features from the $9^{th}$ to the $12^{th}$ layer on the Petspets Dataset, including the associated SC_index for the features, model accuracy, and the L2Norm of the Adapter parameters in the respective layers.
• Figure 2: The relationship between the number of layers for Convpass and RepAdapter and SC_index values on VTAB-1k.
• Figure 3: Overview of our SLS. By aggregating intermediate features output by each layer and calculating the SC_Index of features, SLS dynamically determines the number of layers to prune based on the hyperparameter $\bm{\alpha}$.