Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates

Haoning Xu, Zhaoqing Li, Youjun Chen, Huimeng Wang, Guinan Li, Mengzhe Geng, Chengxi Deng, Xunying Liu

TL;DR

This work tackles the memory and compute demands of SSL speech foundation models by introducing a one-pass compression method that jointly prunes and updates parameters. It leverages sparsity-aware self-pinching gates with a single learnable threshold per layer to generate fine-grained pruning masks, enabling substantial parameter reductions without significantly affecting WER. The method achieves state-of-the-art or competitive WER on LibriSpeech test-clean at high compression (e.g., 7.05% at 4.26x) and demonstrates notable reductions in compression time compared to prior pruning approaches. The approach improves pruning flexibility by enabling mixed sparsity across layers and requires only a small number of additional per-layer parameters, making it practical for on-device deployment and broader applicability in SSL ASR compression.

Abstract

This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.

Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates

TL;DR

This work tackles the memory and compute demands of SSL speech foundation models by introducing a one-pass compression method that jointly prunes and updates parameters. It leverages sparsity-aware self-pinching gates with a single learnable threshold per layer to generate fine-grained pruning masks, enabling substantial parameter reductions without significantly affecting WER. The method achieves state-of-the-art or competitive WER on LibriSpeech test-clean at high compression (e.g., 7.05% at 4.26x) and demonstrates notable reductions in compression time compared to prior pruning approaches. The approach improves pruning flexibility by enabling mixed sparsity across layers and requires only a small number of additional per-layer parameters, making it practical for on-device deployment and broader applicability in SSL ASR compression.

Abstract

This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.

Paper Structure

This paper contains 14 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. wav2vec2.0 and HuBERT Models
  3. Previous works
  4. Uniform Magnitude-based Pruning
  5. NAS-based Channel-wise Pruning
  6. Sparsity-aware Self-pinching Gates
  7. Experiments
  8. Experimental setup
  9. Main results
  10. Comparison with Uniform Magnitude-based Pruning
  11. Comparison with NAS-based Channel-wise Pruning
  12. Comparison with other compression methods
  13. Conclusion
  14. Acknowledgements

Figures (2)

  • Figure 1: Comparison between Uniform Magnitude-based Pruning (UMP), NAS-based Channel-wise Pruning (NAS-CP) and Sparsity-aware Self-pinching Gates (ours). For the $l$-th layer, (a) UMP directly prunes the same proportion of parameters by magnitude across all layers; (b) NAS-CP introduces architecture-dependent parameters proportional to architecture candidates, which are pre-selected before NAS search; (c) Ours utilizes the weights that are already being learned to construct the mask with only one additional threshold.
  • Figure 2: The ASR performance of the pruned wav2vec2-base-100h on the (1) dev and (3) test subsets, as well as the pruned hubert-large on the (2) dev and (4) test subsets with different sparsity using different methods. Abbreviations are the same as those in Figure \ref{['frame']}. Color-matched arrows point to the maximum sparsity preserving lossless compression of the methods in corresponding color.