The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Era

TL;DR

The paper introduces HumDial, a dual-track benchmark for human-like spoken dialogue systems in the LLM era, focusing on emotional intelligence and real-time full-duplex interaction using real-world dialogue data and a hybrid data pipeline. Track I evaluates emotional trajectory, reasoning, and empathetic generation with automated and human assessments, guided by a composite score $Score = 0.2 S_{T1} + 0.2 S_{T2} + 0.1 S_{text} + 0.25 S_{emo} + 0.25 S_{nat}$. Track II evaluates concurrent listening and speaking through interruption and rejection scenarios, quantified by $Score = 0.4 S_{Int} + 0.4 S_{Rej} + 0.2 S_{Delay}$ under standardized GPU-enabled Docker environments. The paper details dataset construction, track configurations, evaluation methodologies, and initial results, highlighting the current strengths and remaining challenges in human-like communication, and sets the stage for broad benchmarking of commercial and open-source Audio-LLMs. The HumDial benchmark aims to push progress toward truly natural, emotionally resonant, and cognitively synchronized spoken dialogue systems with practical impact on user experience.

Abstract

Driven by the rapid advancement of Large Language Models (LLMs), particularly Audio-LLMs and Omni-models, spoken dialogue systems have evolved significantly, progressively narrowing the gap between human-machine and human-human interactions. Achieving truly ``human-like'' communication necessitates a dual capability: emotional intelligence to perceive and resonate with users' emotional states, and robust interaction mechanisms to navigate the dynamic, natural flow of conversation, such as real-time turn-taking. Therefore, we launched the first Human-like Spoken Dialogue Systems Challenge (HumDial) at ICASSP 2026 to benchmark these dual capabilities. Anchored by a sizable dataset derived from authentic human conversations, this initiative establishes a fair evaluation platform across two tracks: (1) Emotional Intelligence, targeting long-term emotion understanding and empathetic generation; and (2) Full-Duplex Interaction, systematically evaluating real-time decision-making under `` listening-while-speaking'' conditions. This paper summarizes the dataset, track configurations, and the final results.