Addressing speaker gender bias in large scale speech translation systems

TL;DR

The paper tackles speaker gender bias in large-scale direct speech translation by first using GPT-4-based reformulation to produce gender-debiased training targets from a manageable data subset, then fine-tuning a direct ST model with a mix of gender-neutral and gender-debiased data. It introduces a three-mode fine-tuning framework (Auto, Masculine, Feminine) and a Gender Representation loss $L_{gr}$ that encourages the encoder to capture gender cues from audio via a combined loss $L_{comb}=\alpha L_{gr}+(1-\alpha)L_{trans}$. Substantial improvements are reported on MuST-SHE for EN→ES/IT, notably increasing Category-1 feminine translation GTA from single-digit percentages to roughly 85–87% while maintaining BLEU, and achieving GTA > 87.5% across speakers with the 3-mode Auto setup; these results surpass state-of-the-art ST systems like Seamless M4T and Canary. The work demonstrates a scalable, cost-effective approach to mitigating speaker gender bias in large ST systems and enables user-preferred gender-form translations, with implications for inclusive and accurate cross-lingual communication.

Abstract

This study addresses the issue of speaker gender bias in Speech Translation (ST) systems, which can lead to offensive and inaccurate translations. The masculine bias often found in large-scale ST systems is typically perpetuated through training data derived from Machine Translation (MT) systems. Our approach involves two key steps. First, we employ Large Language Models (LLMs) to rectify translations based on the speaker's gender in a cost-effective manner. Second, we fine-tune the ST model with the corrected data, enabling the model to generate gender-specific translations directly from audio cues, without the need for explicit gender input. Additionally, we propose a three-mode fine-tuned model for scenarios where the speaker's gender is either predefined or should not be inferred from speech cues. We demonstrate a 70% improvement in translations for female speakers compared to our baseline and other large-scale ST systems, such as Seamless M4T and Canary, on the MuST-SHE test set.

Figures (4)

• Figure 1: Gender de-biasing for large scale training data.
• Figure 2: Prompt to obtain feminine translation form.
• Figure 3: Illustration of model architecture during fine-tuning.
• Figure 4: GTA (%) with and without GR loss for varying $\theta_{neut}$.