Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

Shree Harsha Bokkahalli Satish, Christoph Minixhofer, Maria Teleki, James Caverlee, Ondřej Klejch, Peter Bell, Gustav Eje Henter, Éva Székely

Abstract

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect consistent disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. The bias is implicit: responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, uncovering sharper intersectional disparities.

The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

Abstract

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect consistent disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. The bias is implicit: responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, uncovering sharper intersectional disparities.
Paper Structure (13 sections, 3 figures, 1 table)

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Dataset and Evaluation Setup
  3. Experiments
  4. Preliminary Checks
  5. LLM-as-a-Judge Evaluations
  6. Human Validation of LLM Judge scores
  7. Results
  8. Accent bias in response quality
  9. Intersectional Effects (Accent--Gender)
  10. Results of Human Validation of LLM judge scores
  11. Discussion
  12. Conclusion
  13. Generative AI Use Disclosure

Figures (3)

  • Figure 1: Per-model accent and gender identification accuracy; dashed lines = chance level.
  • Figure 2: Mean helpfulness scores (1--5 Likert scale). Left: by accent and model. Right: by accent and gender (averaged across models $\pm$ Standard Error). Eastern European $\times$ Female is the most disadvantaged subgroup.
  • Figure 3: Left: Proportion of judge LLM BWS selections by accent across four evaluation dimensions. Right: Pairwise helpfulness win rates (%) aggregated across all models. No win rate is over 50% because of ties.