Benchmarking Humans and Machines on Complex Multilingual Speech Understanding Tasks

TL;DR

A systematic paradigm for studying humans and machines in speech question-answering tasks in multilingual settings with clean and mixed-channel speech is proposed and results reveal a key divergence: humans rely on attentional cues that are more streamlined in their native language, whereas LLMs default to parallel information extraction which exceed human skills.

Abstract

Auditory attention and selective phase-locking are central to human speech understanding in complex acoustic scenes and cocktail party settings, yet these capabilities in multilingual subjects remain poorly understood. While machine understanding of natural speech has advanced in recent years, questions persist about comprehension of overlapped and mixed-channel speech. We propose a systematic paradigm for studying humans and machines in speech question-answering tasks in multilingual settings with clean and mixed-channel speech. For human listeners, selective attention to a target speaker was significantly better in their native language (L1) than in their second language (L2). For machine listening, speech-based large language models (LLMs) match or exceed human performance in clean, single-speaker conditions but often struggle to selectively attend in two-speaker settings. These results reveal a key divergence: humans rely on attentional cues that are more streamlined in their native language, whereas LLMs default to parallel information extraction which exceed human skills.

Figures (1)

• Figure 1: Schematic Illustration of the proposed framework for (i) Stimuli recording and pre-processing (Stage 1), (ii) Human evaluation (Stage 2) and (iii) Model evaluation (Stage 3).