SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency

TL;DR

SpeakerSleuth introduces a unified benchmark to evaluate whether Large Audio-Language Models can reliably judge speaker consistency in multi-turn dialogues, using three tasks: Detection, Localization, and Discrimination. The authors construct 1,818 human-validated evaluation instances across four diverse datasets (Bazinga, AMI, Behavior-SD, DailyTalk) spanning 152 speakers, with controlled acoustic variations and voice-conversion scenarios. Nine state-of-the-art LALMs and several speaker-embedding baselines are evaluated, revealing that LALMs exhibit unstable decision thresholds, rely heavily on textual context, and struggle to localize problematic turns, while embedding methods provide stronger, modality-aligned acoustic discrimination. The results highlight a fundamental modality imbalance in current LALMs—textual cues often trump acoustic evidence—emphasizing the need for improved multi-modal integration to build reliable audio-language judges for speaker-consistency verification. Nevertheless, the study shows that LALMs retain intrinsic acoustic discrimination capabilities, suggesting targeted improvements could yield robust, practical evaluation tools for synthesized speech quality and speaker identity across dialogues.

Abstract

Large Audio-Language Models (LALMs) as judges have emerged as a prominent approach for evaluating speech generation quality, yet their ability to assess speaker consistency across multi-turn conversations remains unexplored. We present SpeakerSleuth, a benchmark evaluating whether LALMs can reliably judge speaker consistency in multi-turn dialogues through three tasks reflecting real-world requirements. We construct 1,818 human-verified evaluation instances across four diverse datasets spanning synthetic and real speech, with controlled acoustic difficulty. Evaluating nine widely-used LALMs, we find that models struggle to reliably detect acoustic inconsistencies. For instance, given audio samples of the same speaker's turns, some models overpredict inconsistency, whereas others are overly lenient. Models further struggle to identify the exact turns that are problematic. When other interlocutors' turns are provided together, performance degrades dramatically as models prioritize textual coherence over acoustic cues, failing to detect even obvious gender switches for a speaker. On the other hand, models perform substantially better in choosing the audio that best matches the speaker among several acoustic variants, demonstrating inherent acoustic discrimination capabilities. These findings expose a significant bias in LALMs: they tend to prioritize text over acoustics, revealing fundamental modality imbalances that need to be addressed to build reliable audio-language judges.

Figures (12)

• Figure 1: Overview of SpeakerSleuth.
• Figure 2: SpeakerSleuth Construction Pipeline.
• Figure 3: Distribution of per-sample audio duration across datasets.
• Figure 4: Visualization of the embeddings of dialogue and speaker.
• Figure 5: Audio Data Annotation Tool interface showing the main annotation view.