HydraFormer: One Encoder For All Subsampling Rates

TL;DR

HydraFormer tackles the cost and rigidity of fixed subsampling rates in ASR by unifying multi-rate handling in a single architecture: a shared Conformer encoder (via HydraSub) and a BiTransformer decoder. It employs dynamic subsampling during training, randomly selecting a branch $n$ from $\{4,6,8,\dots,N\}$ and optimizing $\mathcal{L}_{\text{total}} = \alpha \mathcal{L}_{\text{CTC}}(\mathbf{C}_n, \mathbf{T}) + (1-\alpha) \mathcal{L}_{\text{KL}}(\mathbf{B}_n, \mathbf{T})$, with $\alpha \in [0,1]$, updating only the active branch. The approach achieves competitive WER across subsampling rates on AISHELL-1 and LibriSpeech while reducing training and deployment costs to $1/N$ of the expense of training separate rate-specific models, and demonstrates robustness to initialization and the ability to benefit from pretrained single-rate models. These results suggest HydraFormer is practical for diverse real-world deployments, enabling flexible latency/accuracy trade-offs (RTF) without maintaining multiple independent models.

Abstract

In automatic speech recognition, subsampling is essential for tackling diverse scenarios. However, the inadequacy of a single subsampling rate to address various real-world situations often necessitates training and deploying multiple models, consequently increasing associated costs. To address this issue, we propose HydraFormer, comprising HydraSub, a Conformer-based encoder, and a BiTransformer-based decoder. HydraSub encompasses multiple branches, each representing a distinct subsampling rate, allowing for the flexible selection of any branch during inference based on the specific use case. HydraFormer can efficiently manage different subsampling rates, significantly reducing training and deployment expenses. Experiments on AISHELL-1 and LibriSpeech datasets reveal that HydraFormer effectively adapts to various subsampling rates and languages while maintaining high recognition performance. Additionally, HydraFormer showcases exceptional stability, sustaining consistent performance under various initialization conditions, and exhibits robust transferability by learning from pretrained single subsampling rate automatic speech recognition models\footnote{Model code and scripts: https://github.com/HydraFormer/hydraformer}.

Figures (3)

• Figure 1: Comparison of HydraFormer and single subsampling rate ASR models
• Figure 2: Architecture of HydraFormer, where Conv2d (2) denotes convolution layer with stride 2, [PosEnc] denotes optional positional encoding
• Figure 3: Parameter distribution visualization of HydraFormer and baseline models