When Voice Matters: Evidence of Gender Disparity in Positional Bias of SpeechLLMs

TL;DR

The paper addresses gender bias and positional bias in SpeechLLMs by introducing a token-probability analysis of MCQA benchmarks (B1,B2) using a single SpeechLLM, Qwen2-Audio-7B-Instruct, across varying prompts and temperatures. It shows that early-option positional bias persists in SpeechLLMs and interacts with input voice gender, with stronger effects for female voices, even when options are randomized. The authors quantify these effects with token-probabilities and statistical tests, highlighting that few-shot/randomisation strategies may be insufficient to mitigate bias. They conclude that current MCQA-based evaluations fail to account for speech-related confounds and advocate for speech-aware fairness benchmarks to enable trustworthy assessments.

Abstract

The rapid development of SpeechLLM-based conversational AI systems has created a need for robustly benchmarking these efforts, including aspects of fairness and bias. At present, such benchmarks typically rely on multiple choice question answering (MCQA). In this paper, we present the first token-level probabilistic evaluation and response-based study of several issues affecting the use of MCQA in SpeechLLM benchmarking: 1) we examine how model temperature and prompt design affect gender and positional bias on an MCQA gender-bias benchmark; 2) we examine how these biases are affected by the gender of the input voice; and 3) we study to what extent observed trends carry over to a second gender-bias benchmark. Our results show that concerns about positional bias from the text domain are equally valid in the speech domain. We also find the effect to be stronger for female voices than for male voices. To our knowledge, this is the first study to isolate positional bias effects in SpeechLLM-based gender-bias benchmarks. We conclude that current MCQA benchmarks do not account for speech-based bias and alternative strategies are needed to ensure fairness towards all users.

Figures (5)

• Figure 1: Model behaviour on B1 at temperature 1.0, zero-shot prompt, randomised behaviour assignment.
• Figure 2: Average response probability scores vs. temperature when fixing behaviours to different slots on B1 with zero-shot prompt 1 and one-shot prompt 1.
• Figure 3: Average response probability scores vs. temperature when fixing behaviours to different slots on B1 with zero-shot prompt 2 and one-shot prompt 2.
• Figure 4: Anti-Stereotypical slot assignments vs. Selected slot, temperature 1.0.
• Figure 5: Stereotypical slot assignments vs. Selected slot, temperature 1.0.