Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition

Raphaël Bagat, Irina Illina, Emmanuel Vincent

TL;DR

The paper tackles non-native multi-accent ASR by introducing MAS-LoRA, a mixture of accent-specific LoRA experts that are trained per accent and combined at inference without incurring extra cost. The approach supports both accent-agnostic and accent-aware inference, yielding significant WER improvements over regular LoRA and full fine-tuning on the L2-ARCTIC dataset, and even greater gains when the accent is known. MAS-LoRA also demonstrates reduced catastrophic forgetting on native speech, and its accent-aware variant reveals that sharing knowledge across all experts is beneficial, with optimized weighting (β) producing the best results. Overall, MAS-LoRA provides a scalable, parameter-efficient mechanism to robustly handle low-resource, multi-accent ASR with practical implications for international communication domains.

Abstract

We aim to improve the robustness of Automatic Speech Recognition (ASR) systems against non-native speech, particularly in low-resourced multi-accent settings. We introduce Mixture of Accent-Specific LoRAs (MAS-LoRA), a fine-tuning method that leverages a mixture of Low-Rank Adaptation (LoRA) experts, each specialized in a specific accent. This method can be used when the accent is known or unknown at inference time, without the need to fine-tune the model again. Our experiments, conducted using Whisper on the L2-ARCTIC corpus, demonstrate significant improvements in Word Error Rate compared to regular LoRA and full fine-tuning when the accent is unknown. When the accent is known, the results further improve. Furthermore, MAS-LoRA shows less catastrophic forgetting than the other fine-tuning methods. To the best of our knowledge, this is the first use of a mixture of LoRA experts for non-native multi-accent ASR.

Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition

TL;DR

The paper tackles non-native multi-accent ASR by introducing MAS-LoRA, a mixture of accent-specific LoRA experts that are trained per accent and combined at inference without incurring extra cost. The approach supports both accent-agnostic and accent-aware inference, yielding significant WER improvements over regular LoRA and full fine-tuning on the L2-ARCTIC dataset, and even greater gains when the accent is known. MAS-LoRA also demonstrates reduced catastrophic forgetting on native speech, and its accent-aware variant reveals that sharing knowledge across all experts is beneficial, with optimized weighting (β) producing the best results. Overall, MAS-LoRA provides a scalable, parameter-efficient mechanism to robustly handle low-resource, multi-accent ASR with practical implications for international communication domains.

Abstract

We aim to improve the robustness of Automatic Speech Recognition (ASR) systems against non-native speech, particularly in low-resourced multi-accent settings. We introduce Mixture of Accent-Specific LoRAs (MAS-LoRA), a fine-tuning method that leverages a mixture of Low-Rank Adaptation (LoRA) experts, each specialized in a specific accent. This method can be used when the accent is known or unknown at inference time, without the need to fine-tune the model again. Our experiments, conducted using Whisper on the L2-ARCTIC corpus, demonstrate significant improvements in Word Error Rate compared to regular LoRA and full fine-tuning when the accent is unknown. When the accent is known, the results further improve. Furthermore, MAS-LoRA shows less catastrophic forgetting than the other fine-tuning methods. To the best of our knowledge, this is the first use of a mixture of LoRA experts for non-native multi-accent ASR.

Paper Structure

This paper contains 17 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed methodology
  3. Classical LoRA
  4. Mixture of Accent-Specific LoRAs (MAS-LoRA)
  5. Accent-specific fine-tuning
  6. Accent-agnostic inference
  7. Accent-aware inference
  8. Experimental settings
  9. Datasets
  10. General parameters
  11. Evaluation metric
  12. Results and discussions
  13. Baselines
  14. Accent-agnostic MAS-LoRA
  15. Accent-aware MAS-LoRA
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Architecture of MAS-LoRA.
  • Figure 2: Effect of $\beta$ on the WER when the accent label is known at inference. LoRA is applied to both encoder and decoder and MAS-LoRA is applied to the encoder with LoRA in the decoder.