Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EFFUSE: Efficient Self-Supervised Feature Fusion for E2E ASR in Low Resource and Multilingual Scenarios

Tejes Srivastava, Jiatong Shi, William Chen, Shinji Watanabe

TL;DR

EFFUSE addresses the inefficiency of fusing multiple SSL frontends for end-to-end ASR by proposing a two-stage approach: first fuse multiple SSL representations into a shared feature, then train linear predictors to mimic the remaining SSLs from a single primary SSL, allowing inference with only one SSL. The method leverages observed predictive relationships between SSL features (e.g., $R^2 \approx 0.6$), enabling an effective prediction stage that retains fusion performance while dramatically reducing parameters and improving real-time factor. Across low-resource languages (Totonac, Yoloxital Mixtec) and multilingual ML-SUPERB benchmarks, EFFUSE yields substantial performance gains over single-SSL baselines and competitive results with far fewer parameters than full fusion, including instances where prediction variants surpass the fusion topline. The practical impact is improved efficiency for multilingual and low-resource ASR deployments without sacrificing accuracy.

Abstract

Self-Supervised Learning (SSL) models have demonstrated exceptional performance in various speech tasks, particularly in low-resource and multilingual domains. Recent works show that fusing diverse SSL models could achieve superior performance compared to using one SSL model. However, fusing models increases the overall parameter size, leading to higher computational costs. We propose EFFUSE, a novel approach that uses a single SSL model to mimic the features of multiple SSL models via prediction, resulting in a lightweight framework with competitive performance. Our experiments show that EFFUSE outperforms individual SSL models in multilingual speech recognition tasks. Our best performing model achieves an average SUPERB score increase of 63.5 (6.3%) from the SSL baselines in Multilingual Speech Universal PERformance Benchmark (ML-SUPERB), while decreasing parameter size on average by 317M parameters (49%) from the fusion models.

EFFUSE: Efficient Self-Supervised Feature Fusion for E2E ASR in Low Resource and Multilingual Scenarios

TL;DR

EFFUSE addresses the inefficiency of fusing multiple SSL frontends for end-to-end ASR by proposing a two-stage approach: first fuse multiple SSL representations into a shared feature, then train linear predictors to mimic the remaining SSLs from a single primary SSL, allowing inference with only one SSL. The method leverages observed predictive relationships between SSL features (e.g., ), enabling an effective prediction stage that retains fusion performance while dramatically reducing parameters and improving real-time factor. Across low-resource languages (Totonac, Yoloxital Mixtec) and multilingual ML-SUPERB benchmarks, EFFUSE yields substantial performance gains over single-SSL baselines and competitive results with far fewer parameters than full fusion, including instances where prediction variants surpass the fusion topline. The practical impact is improved efficiency for multilingual and low-resource ASR deployments without sacrificing accuracy.

Abstract

Self-Supervised Learning (SSL) models have demonstrated exceptional performance in various speech tasks, particularly in low-resource and multilingual domains. Recent works show that fusing diverse SSL models could achieve superior performance compared to using one SSL model. However, fusing models increases the overall parameter size, leading to higher computational costs. We propose EFFUSE, a novel approach that uses a single SSL model to mimic the features of multiple SSL models via prediction, resulting in a lightweight framework with competitive performance. Our experiments show that EFFUSE outperforms individual SSL models in multilingual speech recognition tasks. Our best performing model achieves an average SUPERB score increase of 63.5 (6.3%) from the SSL baselines in Multilingual Speech Universal PERformance Benchmark (ML-SUPERB), while decreasing parameter size on average by 317M parameters (49%) from the fusion models.
Paper Structure (15 sections, 4 equations, 2 figures, 4 tables)

This paper contains 15 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Predictive Ability Between SSL Models
  3. EFFUSE Formulation
  4. Stage 1: Fusion
  5. Stage 2: Prediction
  6. Experiments
  7. Datasets
  8. Experimental Setup
  9. Results
  10. Low Resource ASR
  11. ML-SUPERB Results
  12. Computational Cost
  13. Conclusion
  14. Acknowledgements
  15. References

Figures (2)

  • Figure 1: The proposed methodology with two SSL Models
  • Figure 2: RTF vs. CER for Totonac. The individual SSL models demonstrate excellent performance in terms of RTF, but there is a trade-off with CER. The prediction models strike a balance by maintaining high performance in both CER and RTF, with additional models included for comparison.