Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark

Dingdong Wang, Jincenzi Wu, Junan Li, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, Helen Meng

TL;DR

This paper introduces MMSU, a large, linguistics-grounded benchmark for spoken language understanding and reasoning, featuring 5,000 expert-annotated MCQs across 47 tasks that span perception and reasoning. It grounds task design in phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics, and uses real-world audio to assess rich acoustic cues. Evaluations across 14 SpeechLLMs reveal a substantial gap to human performance, with semantics being comparatively easier and paralinguistics/phonology remaining particularly challenging. The work provides a rigorous data-construction and human-baseline framework, analyzes model strengths and weaknesses, and proposes a community leaderboard to standardize progress in holistic spoken language understanding.

Abstract

Speech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken language understanding, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio information, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in spoken language. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. To ground our benchmark in linguistic theory, we systematically incorporate a wide range of linguistic phenomena, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics. Through a rigorous evaluation of 14 advanced SpeechLLMs, we identify substantial room for improvement in existing models, highlighting meaningful directions for future optimization. MMSU establishes a new standard for comprehensive assessment of spoken language understanding, providing valuable insights for developing more sophisticated human-AI speech interaction systems. MMSU benchmark is available at https://huggingface.co/datasets/ddwang2000/MMSU. Evaluation Code is available at https://github.com/dingdongwang/MMSU_Bench.

MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark

TL;DR

This paper introduces MMSU, a large, linguistics-grounded benchmark for spoken language understanding and reasoning, featuring 5,000 expert-annotated MCQs across 47 tasks that span perception and reasoning. It grounds task design in phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics, and uses real-world audio to assess rich acoustic cues. Evaluations across 14 SpeechLLMs reveal a substantial gap to human performance, with semantics being comparatively easier and paralinguistics/phonology remaining particularly challenging. The work provides a rigorous data-construction and human-baseline framework, analyzes model strengths and weaknesses, and proposes a community leaderboard to standardize progress in holistic spoken language understanding.

Abstract

Speech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken language understanding, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio information, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in spoken language. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. To ground our benchmark in linguistic theory, we systematically incorporate a wide range of linguistic phenomena, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics. Through a rigorous evaluation of 14 advanced SpeechLLMs, we identify substantial room for improvement in existing models, highlighting meaningful directions for future optimization. MMSU establishes a new standard for comprehensive assessment of spoken language understanding, providing valuable insights for developing more sophisticated human-AI speech interaction systems. MMSU benchmark is available at https://huggingface.co/datasets/ddwang2000/MMSU. Evaluation Code is available at https://github.com/dingdongwang/MMSU_Bench.

Paper Structure

This paper contains 37 sections, 14 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Large Language Models
  4. Benchmarks for SpeechLLMs
  5. MMSU Benchmark
  6. Overview of MMSU
  7. Data Construction
  8. MMSU Statistics
  9. Comparison with Previous Benchmarks
  10. Experiments
  11. Models
  12. Evaluation Strategy
  13. Human Evaluation
  14. Results and Discussion
  15. Main Results
  16. ...and 22 more sections

Figures (14)

  • Figure 1: Overview of the MMSU dataset: MMSU incorporates fine-grained acoustic features, quality assurance through linguistic experts-guided data creation, and tasks across 47 distinct perception and reasoning skills for comprehensive spoken language understanding.
  • Figure 2: Task taxonomy of MMSU. (Left) Distribution of 24 perception-related tasks across linguistics and paralinguistics domains. (Right) Distribution of 23 reasoning tasks across the same domains, forming a comprehensive assessment framework across perception and reasoning abilities.
  • Figure 3: Examples from the MMSU benchmark.
  • Figure 4: Perception related tasks
  • Figure 5: Reasoning related tasks
  • ...and 9 more figures