Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models

Shaojin Ding, David Qiu, David Rim, Yanzhang He, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang, Tara N. Sainath, Zhonglin Han, Jian Li, Amir Yazdanbakhsh, Shivani Agrawal

TL;DR

This work tackles the deployment challenge of large universal speech models (USMs) by proposing a joint quantization and $N$:$M$ sparsity-aware fine-tuning framework for ASR. The approach combines native per-channel quantization with structured sparsity in a prune-and-quantize pipeline and leverages STE for training, evaluated on a 2B-parameter USM backbone with CTC loss. Across ablation and combined experiments, the method achieves a roughly $9.4\%$ of the original model size while incurring only about $7.3\%$ relative WER degradation, with int4 QAT and $2$:$4$ sparsity identified as particularly effective. The study also discusses limitations and future directions, including techniques like variational noise and STE in pruning, aiming to push compression further without sacrificing ASR quality.

Abstract

End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.

USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models

TL;DR

This work tackles the deployment challenge of large universal speech models (USMs) by proposing a joint quantization and : sparsity-aware fine-tuning framework for ASR. The approach combines native per-channel quantization with structured sparsity in a prune-and-quantize pipeline and leverages STE for training, evaluated on a 2B-parameter USM backbone with CTC loss. Across ablation and combined experiments, the method achieves a roughly of the original model size while incurring only about relative WER degradation, with int4 QAT and : sparsity identified as particularly effective. The study also discusses limitations and future directions, including techniques like variational noise and STE in pruning, aiming to push compression further without sacrificing ASR quality.

Abstract

End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.
Paper Structure (17 sections, 4 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 17 sections, 4 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. USM Backbone Architecture
  5. Native Quantization Aware Training (QAT)
  6. Magnitude based Pruning with $N$:$M$ Sparsity
  7. Joint optimization with Quantization and Sparsity
  8. Experimental setup
  9. Datasets
  10. Implementation details
  11. Experiments
  12. Ablation Studies on Quantization
  13. Ablation Studies on Sparsity
  14. Overall Performance of Combining Quantization with Sparsity.
  15. Limitations and Discussions
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Illustration of magnitude based pruning with $N$:$M$ sparsity on a weight matrix. This example has $N=2$ and $M=4$.