Dynamic Quantization Error Propagation in Encoder-Decoder ASR Quantization

TL;DR

This work tackles the challenge of deploying encoder–decoder ASR models on memory-constrained devices by addressing how quantization errors propagate across heterogeneous modules. It introduces FADE, a diagnostic-driven framework that computes per-layer propagation strengths $α_l$ from intrinsic weight vulnerability and calibration reliability, and maps them through a sigmoid to a constrained range, enabling dynamic, fine-grained error correction. The approach improves both mean WER and stability, particularly at 3-bit quantization, across Whisper and Moonshine models and multiple datasets, without requiring extra calibration data or search. Overall, FADE demonstrates that layer-wise, data-driven adjustments to error propagation are crucial for robust low-bit quantization of encoder–decoder ASR systems for edge deployment.

Abstract

Running Automatic Speech Recognition (ASR) models on memory-constrained edge devices requires efficient compression. While layer-wise post-training quantization is effective, it suffers from error accumulation, especially in encoder-decoder architectures. Existing solutions like Quantization Error Propagation (QEP) are suboptimal for ASR due to the model's heterogeneity, processing acoustic features in the encoder while generating text in the decoder. To address this, we propose Fine-grained Alpha for Dynamic Quantization Error Propagation (FADE), which adaptively controls the trade-off between cross-layer error correction and local quantization. Experiments show that FADE significantly improves stability by reducing performance variance across runs, while simultaneously surpassing baselines in mean WER.

Figures (4)

• Figure 1: Visualization of WER v.s. Whisper model size for different quantization techniques. The bubble size represents the standard deviation across runs on 4-bit weight quantization. FADE acquires not only better mean WER but also smaller variance across runs.
• Figure 2: Sensitivity study of $\alpha \in [0, 1]$ for GPTQ+QEP and FADE using 3-bit weight quantization. We use a fixed $\alpha$ across all layers. The Whisper-Tiny model is calibrated on a subset of 128 samples from LibriSpeech-clean and evaluated on 100 samples from LibriSpeech-other. Bubble size and color gradient represent the standard deviation across 3 independent runs.
• Figure 3: Overview of dynamic quantization error propagation in encoder--decoder ASR models.(a) Quantization flow under encoder--decoder asymmetry. The encoder is calibrated using an audio calibration dataset, while the decoder is calibrated using text data together with the quantized encoder outputs. Vanilla calibration quantization refers to calibration-dataset-based PTQ, instantiated as GPTQ gptq, which optimizes the standard objective in Eq. \ref{['eq:ptq_objective']}. To account for error propagation from the encoder to the decoder, corrected quantization introduces a dynamic scaling factor $\alpha_\ell$ to adjust the update direction toward the corrected objective in Eq. \ref{['eq:ptq_objective_corrected']}. The figure provides a schematic view; the same procedure is applied independently within each encoder and decoder block. (b) Dynamic computation of the layer-wise scaling factor $\alpha_\ell$ based on error propagation indicators, enabling adaptive quantization.
• Figure 4: Visualization of average FADE$\alpha_l$ for each Transformer block in quantized Whisper Models on LibriSpeech