Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations

TL;DR

BiasInEar delivers the first systematic examination of speech bias in multilingual multimodal language models by introducing a speech-augmented Benchmark built on Global MMLU Lite, covering English, Chinese, and Korean with balanced gender and accents (70.8 hours, 11,200 questions). It analyzes nine models using accuracy, entropy, APES, and Fleiss' κ to probe linguistic, demographic, and structural perturbations, revealing that option order imposes the strongest robustness degradation, while language and demographic factors exert smaller but notable effects. The study shows that reasoning complexity and pipeline architectures can improve robustness, and that speech tends to amplify biases already present in text-based evaluation, establishing a unified framework for fair, speech-based evaluation of LLMs. By releasing BiasInEar and a systematic methodology, the work bridges text- and speech-based fairness research and provides a resource for future cross-linguistic, cross-modal bias analyses.

Abstract

This work presents the first systematic investigation of speech bias in multilingual MLLMs. We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite, spanning English, Chinese, and Korean, balanced by gender and accent, and totaling 70.8 hours ($\approx$4,249 minutes) of speech with 11,200 questions. Using four complementary metrics (accuracy, entropy, APES, and Fleiss' $κ$), we evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations. Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases. Moreover, architectural design and reasoning strategy substantially affect robustness across languages. Overall, this study establishes a unified framework for assessing fairness and robustness in speech-integrated LLMs, bridging the gap between text- and speech-based evaluation. The resources can be found at https://github.com/ntunlplab/BiasInEar.

Figures (13)

• Figure 1: Overview of this work, which extends question answering from text inputs to multilingual spoken contents across languages, accents, and speakers.
• Figure 2: Mean question entropy across models. Higher entropy indicates greater uncertainty in model predictions.
• Figure 3: Entropy comparison between Culturally Sensitive (CS) and Culturally Agnostic (CA) questions.
• Figure 4: Fleiss' $\kappa$ versus APES across model families. Variables such as language, accent, and gender show higher agreement and stability, while option order yields higher APES and lower $\kappa$, indicating strong sensitivity.
• Figure 5: Fleiss' $\kappa$ versus APES across accent, option order, speed, and volume variables.