Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning

Monorama Swain, Bubai Maji, Jagabandhu Mishra, Markus Schedl, Anders Søgaard, Jesper Rindom Jensen

TL;DR

The paper tackles fairness in English ASR for second-language speakers by evaluating Whisper and Seamless-M4T and documenting large WER disparities across 26 accent groups. It introduces fairness-prompted finetuning with lightweight adapters and a multi-objective loss that combines ERM with spectral decoupling, Group-DRO, and invariant risk minimization, achieving a fused objective that reduces group disparities while preserving accuracy. The results show substantial macro-average WER improvements (relative ~58.7% for Whisper and ~58.5% for Seamless-M4T over the large pretrained models; ~9.7% and ~7.8% over standard ERM baselines), and reveal that larger models mitigate but do not eliminate fairness gaps. The approach demonstrates practical potential for more equitable ASR across diverse L2 accents, although a few accents (e.g., Urdu, Indian English in Seamless) remain challenging, pointing to data representation gaps.

Abstract

In this work, we address the challenge of building fair English ASR systems for second-language speakers. Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups, indicating significant fairness gaps. To mitigate this, we propose fairness-prompted finetuning with lightweight adapters, incorporating Spectral Decoupling (SD), Group Distributionally Robust Optimization (Group-DRO), and Invariant Risk Minimization (IRM). Our proposed fusion of traditional empirical risk minimization (ERM) with cross-entropy and fairness-driven objectives (SD, Group DRO, and IRM) enhances fairness across accent groups while maintaining overall recognition accuracy. In terms of macro-averaged word error rate, our approach achieves a relative improvement of 58.7% and 58.5% over the large pretrained Whisper and SeamlessM4T, and 9.7% and 7.8% over them, finetuning with standard empirical risk minimization with cross-entropy loss.

Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning

TL;DR

The paper tackles fairness in English ASR for second-language speakers by evaluating Whisper and Seamless-M4T and documenting large WER disparities across 26 accent groups. It introduces fairness-prompted finetuning with lightweight adapters and a multi-objective loss that combines ERM with spectral decoupling, Group-DRO, and invariant risk minimization, achieving a fused objective that reduces group disparities while preserving accuracy. The results show substantial macro-average WER improvements (relative ~58.7% for Whisper and ~58.5% for Seamless-M4T over the large pretrained models; ~9.7% and ~7.8% over standard ERM baselines), and reveal that larger models mitigate but do not eliminate fairness gaps. The approach demonstrates practical potential for more equitable ASR across diverse L2 accents, although a few accents (e.g., Urdu, Indian English in Seamless) remain challenging, pointing to data representation gaps.

Abstract

In this work, we address the challenge of building fair English ASR systems for second-language speakers. Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups, indicating significant fairness gaps. To mitigate this, we propose fairness-prompted finetuning with lightweight adapters, incorporating Spectral Decoupling (SD), Group Distributionally Robust Optimization (Group-DRO), and Invariant Risk Minimization (IRM). Our proposed fusion of traditional empirical risk minimization (ERM) with cross-entropy and fairness-driven objectives (SD, Group DRO, and IRM) enhances fairness across accent groups while maintaining overall recognition accuracy. In terms of macro-averaged word error rate, our approach achieves a relative improvement of 58.7% and 58.5% over the large pretrained Whisper and SeamlessM4T, and 9.7% and 7.8% over them, finetuning with standard empirical risk minimization with cross-entropy loss.
Paper Structure (13 sections, 6 equations, 4 figures, 1 table)

This paper contains 13 sections, 6 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Statement
  4. ASR Model
  5. Finetuning with ERM
  6. Fairness-Promoting Finetuning
  7. Experimental Setup and Results
  8. Dataset
  9. Experimental Setup
  10. Fairness-Oriented Evaluation Metrics
  11. Results and Discussion
  12. Analysis
  13. Conclusion

Figures (4)

  • Figure 1: Best WER (%) per accent group for Whisper models before fine-tuning. For each group, the WER shown corresponds to the lowest WER obtained among Whisper variants (i.e., tiny, base, small, medium, large)
  • Figure 2: The fairness-prompting speech recognition system.
  • Figure 3: WER (%) across accent groups for Whisper (left) and Seamless (right) models under different fine-tuning strategies. Without finetuning (blue) show high WER and disparities. ERM (orange) reduces errors but retains variability, while Fusion (green) yields the most balanced profiles, lowering error rates for challenging accents and improving fairness across groups.
  • Figure 4: WER across Whisper models sizes.