Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation

Lina Conti, Dennis Fucci, Marco Gaido, Matteo Negri, Guillaume Wisniewski, Luisa Bentivogli

TL;DR

This work investigates how speech translation models assign gender to speaker-referential terms, revealing that biases arise from broad masculine priors learned in the decoder and data, rather than term-by-term memorization. Using training-data prevalence analyses, ILM approximations, and contrastive feature attribution on spectrograms, the authors show that acoustic input can override ILM biases, and that gender cues are distributed across formants rather than concentrated in pitch. A key mechanism emerges: first-person pronouns act as coreference anchors to the speaker, enabling access to gender information via acoustic cues, especially $F_1$ and $F_2$, with time-localized cues (e.g., “I”) playing a crucial role. These findings have implications for mitigating gender bias in ST, suggesting that interventions must address distributed acoustic cues and coreference-based pathways rather than focusing solely on data rebalancing or pitch manipulation.

Abstract

Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.

Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation

TL;DR

This work investigates how speech translation models assign gender to speaker-referential terms, revealing that biases arise from broad masculine priors learned in the decoder and data, rather than term-by-term memorization. Using training-data prevalence analyses, ILM approximations, and contrastive feature attribution on spectrograms, the authors show that acoustic input can override ILM biases, and that gender cues are distributed across formants rather than concentrated in pitch. A key mechanism emerges: first-person pronouns act as coreference anchors to the speaker, enabling access to gender information via acoustic cues, especially and , with time-localized cues (e.g., “I”) playing a crucial role. These findings have implications for mitigating gender bias in ST, suggesting that interventions must address distributed acoustic cues and coreference-based pathways rather than focusing solely on data rebalancing or pitch manipulation.

Abstract

Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.

Paper Structure

This paper contains 25 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Bias Statement
  3. Related Works
  4. Method
  5. Training Data Prevalence
  6. ILM Approximation
  7. Feature Attribution
  8. Experimental Setup
  9. Data
  10. Models
  11. Training Data Analysis
  12. Internal Language Model Analysis
  13. The Role of Input Audio
  14. Frequency Dimension
  15. Time Dimension
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Example of contrastive feature attribution for the translation of "become" to "diventata"$^F$ instead of "diventato"$^M$. (a) Input spectrogram. (b) Saliency heatmap showing features driving feminine gender assignment. (c) Top 2 % features sufficient to flip gender prediction.
  • Figure 2: Average saliency scores across the frequency dimension for examples that flip, for the Transformer model wang2020fairseq on en$\rightarrow$it translation.