Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PQV-Mobile: A Combined Pruning and Quantization Toolkit to Optimize Vision Transformers for Mobile Applications

Kshitij Bhardwaj

TL;DR

Vision Transformers are memory-intensive, motivating PQV-Mobile to combine pruning and quantization for mobile deployment. The tool supports structured pruning via magnitude, Taylor, and Hessian importance, and FP32→FP16/int8 quantization across backends, demonstrating latency-memory-accuracy trade-offs on DeiT models. A key result is that pruning around 9.375% together with int8 quantization can yield about 7.18x latency reduction with approximately 2.24% accuracy loss, with Taylor pruning performing best among pruning strategies. The end-to-end, deployment-oriented pipeline is open source and compatible with mobile runtime stacks (TorchScript, PyTorch Lite), enabling practical ViT deployment on resource-constrained devices.

Abstract

While Vision Transformers (ViTs) are extremely effective at computer vision tasks and are replacing convolutional neural networks as the new state-of-the-art, they are complex and memory-intensive models. In order to effectively run these models on resource-constrained mobile/edge systems, there is a need to not only compress these models but also to optimize them and convert them into deployment-friendly formats. To this end, this paper presents a combined pruning and quantization tool, called PQV-Mobile, to optimize vision transformers for mobile applications. The tool is able to support different types of structured pruning based on magnitude importance, Taylor importance, and Hessian importance. It also supports quantization from FP32 to FP16 and int8, targeting different mobile hardware backends. We demonstrate the capabilities of our tool and show important latency-memory-accuracy trade-offs for different amounts of pruning and int8 quantization with Facebook Data Efficient Image Transformer (DeiT) models. Our results show that even pruning a DeiT model by 9.375% and quantizing it to int8 from FP32 followed by optimizing for mobile applications, we find a latency reduction by 7.18X with a small accuracy loss of 2.24%. The tool is open source.

PQV-Mobile: A Combined Pruning and Quantization Toolkit to Optimize Vision Transformers for Mobile Applications

TL;DR

Vision Transformers are memory-intensive, motivating PQV-Mobile to combine pruning and quantization for mobile deployment. The tool supports structured pruning via magnitude, Taylor, and Hessian importance, and FP32→FP16/int8 quantization across backends, demonstrating latency-memory-accuracy trade-offs on DeiT models. A key result is that pruning around 9.375% together with int8 quantization can yield about 7.18x latency reduction with approximately 2.24% accuracy loss, with Taylor pruning performing best among pruning strategies. The end-to-end, deployment-oriented pipeline is open source and compatible with mobile runtime stacks (TorchScript, PyTorch Lite), enabling practical ViT deployment on resource-constrained devices.

Abstract

While Vision Transformers (ViTs) are extremely effective at computer vision tasks and are replacing convolutional neural networks as the new state-of-the-art, they are complex and memory-intensive models. In order to effectively run these models on resource-constrained mobile/edge systems, there is a need to not only compress these models but also to optimize them and convert them into deployment-friendly formats. To this end, this paper presents a combined pruning and quantization tool, called PQV-Mobile, to optimize vision transformers for mobile applications. The tool is able to support different types of structured pruning based on magnitude importance, Taylor importance, and Hessian importance. It also supports quantization from FP32 to FP16 and int8, targeting different mobile hardware backends. We demonstrate the capabilities of our tool and show important latency-memory-accuracy trade-offs for different amounts of pruning and int8 quantization with Facebook Data Efficient Image Transformer (DeiT) models. Our results show that even pruning a DeiT model by 9.375% and quantizing it to int8 from FP32 followed by optimizing for mobile applications, we find a latency reduction by 7.18X with a small accuracy loss of 2.24%. The tool is open source.
Paper Structure (7 sections, 7 figures)

This paper contains 7 sections, 7 figures.

Table of Contents

  1. Introduction
  2. PQV-Mobile Tool
  3. Pruning method
  4. Quantization method
  5. Experimental Results
  6. Conclusion and Future Work
  7. Acknowledgements

Figures (7)

  • Figure 1: PQV-Mobile tool flow
  • Figure 2: Latency and memory results for deit_base_patch16 model with varying pruning and quantization. All models are scripted, mobile optimized, and converted to Pytorch Lite format.
  • Figure 3: Latency profile for deit3_medium_patch16 int8 model at 9.375% pruning level for x86 backend. The model is scripted, mobile optimized, and converted to Pytorch Lite format.
  • Figure 4: Accuracy results for deit_base_patch16 model with varying pruning and quantization. All models are scripted, mobile optimized, and converted to Pytorch Lite format.
  • Figure 5: Accuracy results for deit_base_patch16 model with different structured pruning groupings (for both FP32 and int8 models). All models are scripted, mobile optimized, and converted to Pytorch Lite format.
  • ...and 2 more figures