Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective

TL;DR

This research introduces a novel data-model co-design perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner and finds that subnetworks discovered by VPNs from pre-trained models enjoy better transferability across diverse downstream scenarios.

Abstract

The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as the product, have demonstrated numerous favorable benefits like low complexity, undamaged generalization, etc. Most of the prominent pruning strategies are invented from a model-centric perspective, focusing on searching and preserving crucial weights by analyzing network topologies. However, the role of data and its interplay with model-centric pruning has remained relatively unexplored. In this research, we introduce a novel data-model co-design perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner. Specifically, customized Visual Prompts are mounted to upgrade neural Network sparsification in our proposed VPNs framework. As a pioneering effort, this paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach. Extensive experiments with 3 network architectures and 8 datasets evidence the substantial performance improvements from VPNs over existing start-of-the-art pruning algorithms. Furthermore, we find that subnetworks discovered by VPNs from pre-trained models enjoy better transferability across diverse downstream scenarios. These insights shed light on new promising possibilities of data-model co-designs for vision model sparsification.

Figures (11)

• Figure 1: Post-pruning Prompt Results. Performance of $5$ pruning methods and their post-pruning prompt counterparts on ResNet-$18$ and CIFAR$10$, which are marked as $\bullet$ and $\bigstar$, respectively. The dashed line indicates the dense network's performance. (a) Post-pruning with zero-shot. (b) Post-pruning with fine-tuning. Post-pruning prompt is only valid without fine-tuning.
• Figure 2: Overview of VPNs. In stage $1$, it locates superior sparse topologies from a data-model perspective. A tailored VP is added to input samples and weight masks are jointly optimized together with the VP. In stage $2$, the identified subnetwork is further fine-tuned with its VP.
• Figure 3: Our VP.
• Figure 4: Downstream Fine-tuning Results. The performance overview of $9$ unstructured pruning algorithms. All the models are pre-trained on ImageNet-1K; and then pruned and fine-tuned both on the specific downstream dataset. The performance of the dense model and VPNs' best are marked using dashed lines. All the results are averaged over 3 runs. VPNs consistently outperforms other baselines on all eight tasks.
• Figure 5: Downstream Fine-tuning Results. The performance overview of VPNs, HYDRA, and OMP. All the results are obtained with ImageNet-$1$K pre-trained ResNet-$50$ and VGG-$16$, fine-tuned on Tiny-ImageNet. VPNs consistently has superior performance.