Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting

Jing Xu, Minglin Wu, Xueyuan Chen, Xixin Wu, Helen Meng

TL;DR

MiLorE-SSL tackles the problem of scaling multilingual self-supervised speech models without catastrophic forgetting. It proposes a parameter-efficient approach that combines LoRA-based experts with a soft mixture-of-experts router, integrated into a HuBERT-based SSL backbone where the FFN is frozen and only the LoRA modules and router are trained. A limited replay strategy using a small set of data from existing languages mitigates forgetting while avoiding full historical corpora. Experiments on ML-SUPERB show that MiLorE-SSL improves English, Mandarin, and Cantonese ASR and LID performance with only 2.14% trainable parameters, outperforming strong baselines and demonstrating robustness to out-of-domain data, with replay data as little as 100 hours. The approach highlights a viable path for scalable, continual multilingual SSL learning with flexible cross-language sharing and minimal resource overhead.

Abstract

Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new languages is computationally expensive, while sequential training without migitation strategies often leads to catastrophic forgetting. To address this, we propose MiLorE-SSL, a lightweight framework that combines LoRA modules with a soft mixture-of-experts (MoE) mechanism for efficient continual multilingual training. LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference. To further mitigate forgetting, we introduce limited replay data from existing languages, avoiding reliance on large historical corpora. Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters.

MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting

TL;DR

MiLorE-SSL tackles the problem of scaling multilingual self-supervised speech models without catastrophic forgetting. It proposes a parameter-efficient approach that combines LoRA-based experts with a soft mixture-of-experts router, integrated into a HuBERT-based SSL backbone where the FFN is frozen and only the LoRA modules and router are trained. A limited replay strategy using a small set of data from existing languages mitigates forgetting while avoiding full historical corpora. Experiments on ML-SUPERB show that MiLorE-SSL improves English, Mandarin, and Cantonese ASR and LID performance with only 2.14% trainable parameters, outperforming strong baselines and demonstrating robustness to out-of-domain data, with replay data as little as 100 hours. The approach highlights a viable path for scalable, continual multilingual SSL learning with flexible cross-language sharing and minimal resource overhead.

Abstract

Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new languages is computationally expensive, while sequential training without migitation strategies often leads to catastrophic forgetting. To address this, we propose MiLorE-SSL, a lightweight framework that combines LoRA modules with a soft mixture-of-experts (MoE) mechanism for efficient continual multilingual training. LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference. To further mitigate forgetting, we introduce limited replay data from existing languages, avoiding reliance on large historical corpora. Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters.
Paper Structure (15 sections, 5 equations, 2 figures, 4 tables)

This paper contains 15 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. MiLorE-SSL architecture
  4. Replay strategy
  5. Training strategy
  6. Experimental Setup
  7. Datasets
  8. Training configurations
  9. Evaluation configuration
  10. Experimental results
  11. Main results
  12. Analysis of Expert activation patterns
  13. Ablation Study
  14. Conclusion
  15. Acknowledgement

Figures (2)

  • Figure 1: Overview of MiLorE-SSL framework. (a) Architecture of HuBERT-based SSL models. (b) Transformer block with MiLorE module. (c) Architecture of MiLorE module, where a router selects experts to process input hidden states alongside a frozen FFN.
  • Figure 2: Layer-wise expert weights across languages