Benchmarking Speech Systems for Frontline Health Conversations: The DISPLACE-M Challenge

Abstract

The DIarization and Speech Processing for LAnguage understanding in Conversational Environments - Medical (DISPLACE-M) challenge introduces a conversational AI benchmark focused on understanding goal-oriented, real-world medical dialogues collected in the field. The challenge addresses multi-speaker interactions between healthcare workers and seekers characterized by spontaneous, noisy and overlapping speech across Indian languages and dialects. As part of the challenge, medical conversational dataset comprising 25 hours of development data and 10 hours of blind evaluation recordings was released. We provided baseline systems within a unified end-to-end pipeline across 4 tasks - speaker diarization, automatic speech recognition, topic identification and dialogue summarization - to enable consistent benchmarking. System performance is evaluated using established metrics such as diarization error rate (DER), time-constrained minimum-permutation word error rate (tcpWER), and ROUGE-L. During this evaluation (Phase-I), 12 teams, across the globe, actively participated pushing the baseline systems on these metrics. However, even with a 6-8 week dedicated effort from various participants, the task is shown to be substantially challenging, and the existing systems are significantly short of healthcare deployment readiness.

Figures (3)

• Figure 1: Timeline of DISPLACE-M
• Figure 2: DISPLACE-M Baseline system development pipeline which involves a cascaded approach of speaker diarization (SD), ASR and topic identification / summarization modules.
• Figure 3: Comparison of Diarization Error Rate (DER %) for Track 1 Evaluation Phase-I across submitted teams.