Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF

Xingchen Song, Di Wu, Binbin Zhang, Dinghao Zhou, Zhendong Peng, Bo Dang, Fuping Pan, Chao Yang

TL;DR

To scale speech foundation models, the paper investigates sparse Mixture-of-Experts (MoE) layers as an energy-efficient alternative to dense FFN layers in ASR. It shows that replacing all FFN modules with MoE across both the Conformer encoder and Transformer decoder, trained under the unified U2++ two-pass CTC/AED framework, yields Dense-1B-level accuracy while keeping Dense-225M-level Real Time Factor and enabling streaming decoding. A key contribution is demonstrating streaming-capable MoE models without auxiliary routing losses or extra embeddings, achieving competitive WER with minimal latency overhead. The results support a practical path to scaling ASR models that balances accuracy, latency, and deployment feasibility.

Abstract

Scale has opened new frontiers in natural language processing, but at a high cost. In response, by learning to only activate a subset of parameters in training and inference, Mixture-of-Experts (MoE) have been proposed as an energy efficient path to even larger and more capable language models and this shift towards a new generation of foundation models is gaining momentum, particularly within the field of Automatic Speech Recognition (ASR). Recent works that incorporating MoE into ASR models have complex designs such as routing frames via supplementary embedding network, improving multilingual ability for the experts, and utilizing dedicated auxiliary losses for either expert load balancing or specific language handling. We found that delicate designs are not necessary, while an embarrassingly simple substitution of MoE layers for all Feed-Forward Network (FFN) layers is competent for the ASR task. To be more specific, we benchmark our proposed model on a large scale inner-source dataset (160k hours), the results show that we can scale our baseline Conformer (Dense-225M) to its MoE counterparts (MoE-1B) and achieve Dense-1B level Word Error Rate (WER) while maintaining a Dense-225M level Real Time Factor (RTF). Furthermore, by applying Unified 2-pass framework with bidirectional attention decoders (U2++), we achieve the streaming and non-streaming decoding modes in a single MoE based model, which we call U2++ MoE. We hope that our study can facilitate the research on scaling speech foundation models without sacrificing deployment efficiency.

U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF

TL;DR

To scale speech foundation models, the paper investigates sparse Mixture-of-Experts (MoE) layers as an energy-efficient alternative to dense FFN layers in ASR. It shows that replacing all FFN modules with MoE across both the Conformer encoder and Transformer decoder, trained under the unified U2++ two-pass CTC/AED framework, yields Dense-1B-level accuracy while keeping Dense-225M-level Real Time Factor and enabling streaming decoding. A key contribution is demonstrating streaming-capable MoE models without auxiliary routing losses or extra embeddings, achieving competitive WER with minimal latency overhead. The results support a practical path to scaling ASR models that balances accuracy, latency, and deployment feasibility.

Abstract

Scale has opened new frontiers in natural language processing, but at a high cost. In response, by learning to only activate a subset of parameters in training and inference, Mixture-of-Experts (MoE) have been proposed as an energy efficient path to even larger and more capable language models and this shift towards a new generation of foundation models is gaining momentum, particularly within the field of Automatic Speech Recognition (ASR). Recent works that incorporating MoE into ASR models have complex designs such as routing frames via supplementary embedding network, improving multilingual ability for the experts, and utilizing dedicated auxiliary losses for either expert load balancing or specific language handling. We found that delicate designs are not necessary, while an embarrassingly simple substitution of MoE layers for all Feed-Forward Network (FFN) layers is competent for the ASR task. To be more specific, we benchmark our proposed model on a large scale inner-source dataset (160k hours), the results show that we can scale our baseline Conformer (Dense-225M) to its MoE counterparts (MoE-1B) and achieve Dense-1B level Word Error Rate (WER) while maintaining a Dense-225M level Real Time Factor (RTF). Furthermore, by applying Unified 2-pass framework with bidirectional attention decoders (U2++), we achieve the streaming and non-streaming decoding modes in a single MoE based model, which we call U2++ MoE. We hope that our study can facilitate the research on scaling speech foundation models without sacrificing deployment efficiency.
Paper Structure (10 sections, 1 equation, 1 figure, 4 tables)

This paper contains 10 sections, 1 equation, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Experiments
  5. Datasets
  6. Training Details
  7. Main Results on 160k hours
  8. Streaming Capability
  9. Conclusion
  10. Acknowledgements

Figures (1)

  • Figure 1: The proposed U2++ MoE, a unified (streaming and non-streaming) two-pass (encoder for 1st pass decoding and decoder for 2nd pass rescoring) joint CTC/AED framework, enhanced with bidirectional decoders and Mixture-of-Experts. For efficient compression of speech frames, we employ 1/8 subsampling and structure our architecture with $M$ encoder layers alongside $2N$ decoder layers, wherein equal divisions of $N$ layers are allocated to both the right-to-left and left-to-right decoders.