Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models

TL;DR

This work targets memory and compute bottlenecks of Whisper-based ASR for edge deployment by introducing TSPAR, a framework that combines Sparse Group LASSO–based structured sparsity with a non-iterative, weight-statistics–driven pruning strategy. The method fine-tunes Whisper models with a regularized loss to induce column-wise sparsity and then performs a two-pass pruning to remove approximately sparse columns followed by residual weight pruning, all without adapters. A custom Hindi text normalizer is also designed to enable fair WER evaluation across languages. Empirically, TSPAR achieves substantial memory and FLOPs reductions while maintaining, or even improving, WER on Hindi ASR compared to Iterative Magnitude Pruning, making compact Whisper variants viable for resource-constrained edge devices.

Abstract

Whisper models have achieved remarkable progress in speech recognition; yet their large size remains a bottleneck for deployment on resource-constrained edge devices. This paper proposes a framework to design fine-tuned variants of Whisper which address the above problem. Structured sparsity is enforced via the Sparse Group LASSO penalty as a loss regularizer, to reduce the number of FLOating Point operations (FLOPs). Further, a weight statistics aware pruning algorithm is proposed. We also design our custom text normalizer for WER evaluation. On Common Voice 11.0 Hindi dataset, we obtain, without degrading WER, (a) 35.4% reduction in model parameters, 14.25% lower memory consumption and 18.5% fewer FLOPs on Whisper-small, and (b) 31% reduction in model parameters, 15.29% lower memory consumption and 16.95% fewer FLOPs on Whisper-medium; and, (c) substantially outperform the state-of-the-art Iterative Magnitude Pruning based method by pruning 18.7% more parameters along with a 12.31 reduction in WER.

Figures (1)

• Figure 1: Different types of weight distributions