Uncovering the Hidden Cost of Model Compression

TL;DR

The fact that model compression detrimentally impacts the performance of visual prompting-based transfer is shed, particularly evident in scenarios with low data volume, and it is illustrated that such negative effects on calibration are not present when models are compressed via quantization.

Abstract

In an age dominated by resource-intensive foundation models, the ability to efficiently adapt to downstream tasks is crucial. Visual Prompting (VP), drawing inspiration from the prompting techniques employed in Large Language Models (LLMs), has emerged as a pivotal method for transfer learning in the realm of computer vision. As the importance of efficiency continues to rise, research into model compression has become indispensable in alleviating the computational burdens associated with training and deploying over-parameterized neural networks. A primary objective in model compression is to develop sparse and/or quantized models capable of matching or even surpassing the performance of their over-parameterized, full-precision counterparts. Although previous studies have explored the effects of model compression on transfer learning, its impact on visual prompting-based transfer remains unclear. This study aims to bridge this gap, shedding light on the fact that model compression detrimentally impacts the performance of visual prompting-based transfer, particularly evident in scenarios with low data volume. Furthermore, our findings underscore the adverse influence of sparsity on the calibration of downstream visual-prompted models. However, intriguingly, we also illustrate that such negative effects on calibration are not present when models are compressed via quantization. This empirical investigation underscores the need for a nuanced understanding beyond mere accuracy in sparse and quantized settings, thereby paving the way for further exploration in Visual Prompting techniques tailored for sparse and quantized models.

Figures (25)

• Figure 1: When examining the label mapping chen2023understanding of the ResNet-50 Sparse LT model he2016deepfrankle2018lottery alongside its dense counterpart across target classes within the OxfordPetsparkhi12a and DTDcimpoi14describing datasets, a notable distinction emerges: the dense model exhibits a more semantically accurate label mapping. In contrast, the sparse variant often assigns target classes to unrelated classes from the source dataset. This trend echoes similarly in the context of quantization, where the full-precision DeiT (32-bit) touvron2020training demonstrates superior semantic accuracy and consistency in label mapping compared to its quantized counterpart (2-bit) huang2023variation across various target classes within the OxfordPets and DTD datasets.
• Figure 2: GMP-pruned ResNet-18/34. Transfer performance measured by test accuracy of pruned ResNet-18/34 model on a variety of downstream datasets and varying levels of data budgets.
• Figure 3: AC/DC-pruned ResNet-50. Transfer performance measured by test accuracy of pruned ResNet-50 model on a variety of downstream datasets and varying levels of data budgets.
• Figure 4: RigL-pruned ResNet-50. Transfer performance measured by test accuracy of pruned ResNet-50 model on a variety of downstream datasets and varying levels of data budgets.
• Figure 5: VVTQuantized DeiT-T. Transfer performance measured by test accuracy of quantized DeiT-T models on a variety of downstream datasets and varying levels of data budgets.