Enhanced Structured Lasso Pruning with Class-wise Information

TL;DR

This work tackles pruning CNNs for edge deployment by incorporating precise class-wise information into structured sparsity. It introduces two adaptive pruning schemes, sGLP-IB and sTLP-IB, that couple Information Bottleneck theory with graph- and tree-structured Lasso, operating in a layer-wise fashion and using Gram-matrix representations to reduce computation. The approach yields state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet across multiple architectures, with substantial reductions in parameters and FLOPs and minimal accuracy loss (e.g., Top-1 ~76.12% on ImageNet with ~57% FLOPs pruning and ~55% parameters pruning, 0.03% drop). A convex optimization framework with proximal gradient descent and an adaptive lambda strategy ensures convergence and scalable pruning, making the method practical for resource-constrained devices.

Abstract

Modern applications require lightweight neural network models. Most existing neural network pruning methods focus on removing unimportant filters; however, these may result in the loss of statistical information after pruning due to failing to consider the class-wise information. In this paper, we employ the structured lasso from the perspective of utilizing precise class-wise information for model pruning with the help of Information Bottleneck theory, which guides us to ensure the retention of statistical information before and after pruning. With these techniques, we propose two novel adaptive network pruning schemes in parallel: sparse graph-structured lasso pruning with Information Bottleneck (sGLP-IB) and sparse tree-guided lasso pruning with Information Bottleneck (sTLP-IB). The key component is that we prune the model filters utilizing sGLP-IB and sTLP-IB with more precise structured class-wise relatedness. Compared to multiple state-of-the-art methods, our approaches achieve the best performance across three datasets and six model structures on extensive experiments. For example, with the VGG16 model based on the CIFAR-10 dataset, we can reduce the parameters by 85%, decrease the FLOPs by 61%, and maintain an accuracy of 94.10% (0.14% better than the original). For large-scale ImageNet, we can reduce the parameters by 55% while keeping the accuracy at 76.12% (only drop 0.03%) using the ResNet architecture. In summary, we succeed in reducing the model size and computational resource usage while maintaining the effectiveness of accuracy.

Figures (3)

• Figure 1: The overview of our methods. To utilize information bottleneck theory, ① reshape the input and output feature maps. ② transforms the reshaped feature maps with Gram matrix. Then, we utilize the structured lasso to account for the structured class-wise information to get the linkage matrix and prune the filters.
• Figure 2: t-SNE visualizations of pruned models with different kernel functions before finetuning using VGGNet-16 on CIFAR-10 (T denotes sTLP-IB and G denotes sGLP-IB).
• Figure 3: Our example experimental prototype for resource-constrained devices.