HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition

TL;DR

HuBERT-EE tackles the high computational cost of self-supervised ASR by introducing dynamic early exiting through self-attention-based branches inserted at intermediate layers of HuBERT-base. The method jointly fine-tunes the final output and all early exits (with a single loss weight) and uses entropy-based exiting to decide when to stop, achieving faster inference with controlled degradation in WER. Empirical results on LibriSpeech show HuBERT-EE outperforming conventional compression methods in speed-accuracy trade-offs and enabling flexible speed adjustments without retraining. This approach provides a practical pathway to deploy large SSL ASR models in resource-constrained or latency-sensitive environments, while maintaining competitive recognition performance.

Abstract

Pre-training with self-supervised models, such as Hidden-unit BERT (HuBERT) and wav2vec 2.0, has brought significant improvements in automatic speech recognition (ASR). However, these models usually require an expensive computational cost to achieve outstanding performance, slowing down the inference speed. To improve the model efficiency, we introduce an early exit scheme for ASR, namely HuBERT-EE, that allows the model to stop the inference dynamically. In HuBERT-EE, multiple early exit branches are added at the intermediate layers. When the intermediate prediction of the early exit branch is confident, the model stops the inference, and the corresponding result can be returned early. We investigate the proper early exiting criterion and fine-tuning strategy to effectively perform early exiting. Experimental results on the LibriSpeech show that HuBERT-EE can accelerate the inference of the HuBERT while simultaneously balancing the trade-off between the performance and the latency.

Figures (4)

• Figure 1: Overview of the HuBERT-EE. In contrast to the original HuBERT model, our proposed approach can stop the inference dynamically. If an early exit branch is sufficiently confident in its prediction, the corresponding result can be exited early.
• Figure 2: Quality–efficiency trade-offs on test-clean dataset using different fine-tuning strategies for HuBERT-EE. We set entropy thresholds $S$ from 0.008 to 0.002.
• Figure 3: Performance comparison on test-clean. All results were evaluated based on greedy decoding. We set different thresholds $S$ from 0.005 to 0.002 for HuBERT-EE. The proposed model was fine-tuned with joint training.
• Figure 4: The number of exiting samples on text-clean. Samples that did not exit earlier were returned via the last linear layer.