Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, Changouk Kim

TL;DR

M3-SLU introduces a targeted benchmark for speaker-attributed reasoning in multimodal large language models by combining four public multi-speaker corpora into 12,873 validated segments. It defines two tasks—Speaker-Attributed QA and Utterance Matching—and deploys a four-stage data construction pipeline with both automated GPT-4o-based generation/validation and final human checks, supplemented by an LLM-as-Judge evaluation framework. Across SDR and end-to-end MLLM experiments, results show models can extract what was said but struggle to correctly identify who spoke when, highlighting a persistent gap in speaker-grounded dialogue understanding. The benchmark provides a realistic, long-form multi-speaker evaluation platform to guide future work on speaker roles, turn-taking, and dialogue grounding in multimodal systems, moving beyond transcription accuracy toward true speaker-aware reasoning.

Abstract

We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding.

M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

TL;DR

M3-SLU introduces a targeted benchmark for speaker-attributed reasoning in multimodal large language models by combining four public multi-speaker corpora into 12,873 validated segments. It defines two tasks—Speaker-Attributed QA and Utterance Matching—and deploys a four-stage data construction pipeline with both automated GPT-4o-based generation/validation and final human checks, supplemented by an LLM-as-Judge evaluation framework. Across SDR and end-to-end MLLM experiments, results show models can extract what was said but struggle to correctly identify who spoke when, highlighting a persistent gap in speaker-grounded dialogue understanding. The benchmark provides a realistic, long-form multi-speaker evaluation platform to guide future work on speaker roles, turn-taking, and dialogue grounding in multimodal systems, moving beyond transcription accuracy toward true speaker-aware reasoning.

Abstract

We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding.
Paper Structure (33 sections, 1 equation, 5 figures, 7 tables)

This paper contains 33 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Understanding Models
  4. Speech Understanding Benchmarks
  5. Multi-speaker Speech Understanding Benchmarks
  6. M3-SLU Benchmark
  7. Purpose of Benchmarks
  8. Overview of Benchmarks
  9. Benchmark Construction Pipeline
  10. Stage 1. Speech Data Pre-processing
  11. Stage 2. Automated Data Curation Loop
  12. Generation.
  13. Validation and Refinement in Task 1.
  14. Validation and Refinement in Task 2.
  15. Stage 3. Data Integration
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Overview of the M3-SLU Benchmark.
  • Figure 2: Example of M3-SLU Benchmark. Although they are expressed in text form, they are all voice conversation datasets. The generated Question and GT Answer are in text format.
  • Figure 3: The 4-Stage Hybrid Pipeline for M3-SLU Benchmark Construction.
  • Figure 4: Detailed Pipeline for Validation and Refinement in Task 1(QA)
  • Figure 5: Detailed Pipeline for Validation and Refinement in Task 2(T/F)