Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond

Mardhiyah Sanni, Tassallah Abdullahi, Devendra D. Kayande, Emmanuel Ayodele, Naome A. Etori, Michael S. Mollel, Moshood Yekini, Chibuzor Okocha, Lukman E. Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji

TL;DR

Afrispeech-Dialog tackles the lack of African-accented English conversational data for ASR and downstream clinical tasks by introducing a 7-hour, 50-conversation benchmark spanning medical and general domains. The work benchmarks state-of-the-art diarization and ASR models and investigates LLM-based medical summarization using both human and machine transcripts, revealing notable degradation when processing accented speech and highlighting cascading-error effects. It demonstrates that while some LLMs can recall key clinical details, performance is inconsistent across models and input quality, underscoring the need for inclusive, high-quality data and robust pipelines. Overall, the dataset provides a foundational resource to advance diarization, ASR, and clinical NLP in low-resource African contexts and informs future development toward more equitable conversational AI systems.

Abstract

Speech technologies are transforming interactions across various sectors, from healthcare to call centers and robots, yet their performance on African-accented conversations remains underexplored. We introduce Afrispeech-Dialog, a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations, designed to evaluate automatic speech recognition (ASR) and related technologies. We assess state-of-the-art (SOTA) speaker diarization and ASR systems on long-form, accented speech, comparing their performance with native accents and discover a 10%+ performance degradation. Additionally, we explore medical conversation summarization capabilities of large language models (LLMs) to demonstrate the impact of ASR errors on downstream medical summaries, providing insights into the challenges and opportunities for speech technologies in the Global South. Our work highlights the need for more inclusive datasets to advance conversational AI in low-resource settings.

Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond

TL;DR

Afrispeech-Dialog tackles the lack of African-accented English conversational data for ASR and downstream clinical tasks by introducing a 7-hour, 50-conversation benchmark spanning medical and general domains. The work benchmarks state-of-the-art diarization and ASR models and investigates LLM-based medical summarization using both human and machine transcripts, revealing notable degradation when processing accented speech and highlighting cascading-error effects. It demonstrates that while some LLMs can recall key clinical details, performance is inconsistent across models and input quality, underscoring the need for inclusive, high-quality data and robust pipelines. Overall, the dataset provides a foundational resource to advance diarization, ASR, and clinical NLP in low-resource African contexts and informs future development toward more equitable conversational AI systems.

Abstract

Speech technologies are transforming interactions across various sectors, from healthcare to call centers and robots, yet their performance on African-accented conversations remains underexplored. We introduce Afrispeech-Dialog, a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations, designed to evaluate automatic speech recognition (ASR) and related technologies. We assess state-of-the-art (SOTA) speaker diarization and ASR systems on long-form, accented speech, comparing their performance with native accents and discover a 10%+ performance degradation. Additionally, we explore medical conversation summarization capabilities of large language models (LLMs) to demonstrate the impact of ASR errors on downstream medical summaries, providing insights into the challenges and opportunities for speech technologies in the Global South. Our work highlights the need for more inclusive datasets to advance conversational AI in low-resource settings.

Paper Structure

This paper contains 46 sections, 2 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ASR in Medical Conversations and Summarization
  4. Non-medical conversational ASR
  5. African Accented ASR
  6. Speaker Diarization in Multi-Speaker Conversations
  7. Methodology
  8. Dataset Statistics
  9. Collecting Conversations
  10. Recording Characteristics
  11. Transcription Process
  12. Speaker Diarization
  13. Evaluation Metrics
  14. Automatic Speech Recognition (ASR)
  15. Preprocessing
  16. ...and 31 more sections

Figures (5)

  • Figure 1: AfriSpeech Dialog: Dataset and Benchmarking Pipeline
  • Figure 2: Comparison of Medical and General DER for Different Models
  • Figure 3: Comparison of Medical and General WER for Different Models
  • Figure 4: Summarization results for several LLMs
  • Figure 5: WER, Med WER, and Non-Med WER for Various Models