The Multicultural Medical Assistant: Can LLMs Improve Medical ASR Errors Across Borders?

TL;DR

This work tackles the problem of ASR errors in medical transcription across borders by evaluating six ASR systems and three LLMs on 191 conversations from Nigeria, the UK, and the US. It introduces a reproducible, end-to-end pipeline that includes punctuation, diarization, and correction steps, with medical concepts extracted via Google Healthcare NLP and evaluated using WER and MC-WER. The study finds substantial cross-regional differences in transcription accuracy, demonstrates that LLM diarization can outperform native ASR diarization, and shows that LLM corrections are most helpful for lower-performing ASR systems while offering limited gains for high-performing models. These findings illuminate both the potential and current limits of generic LLMs for improving global medical transcription, and they underscore the need for accent-aware training data and clinically meaningful evaluation metrics.

Abstract

The global adoption of Large Language Models (LLMs) in healthcare shows promise to enhance clinical workflows and improve patient outcomes. However, Automatic Speech Recognition (ASR) errors in critical medical terms remain a significant challenge. These errors can compromise patient care and safety if not detected. This study investigates the prevalence and impact of ASR errors in medical transcription in Nigeria, the United Kingdom, and the United States. By evaluating raw and LLM-corrected transcriptions of accented English in these regions, we assess the potential and limitations of LLMs to address challenges related to accents and medical terminology in ASR. Our findings highlight significant disparities in ASR accuracy across regions and identify specific conditions under which LLM corrections are most effective.

Figures (8)

• Figure 1: The steps of our Chain-of-Thought (CoT) prompting pipeline for medical conversation processing.
• Figure 2: Distribution of WER across baseline ASR transcriptions for all speakers in each dataset.
• Figure 3: Distribution of MC-WER across baseline ASR transcriptions for all speakers in each dataset.
• Figure 4: WER Distributions for patient and doctor speech for the baseline ASR system, Soniox (orange), and top performing LLM and ASR pairs.
• Figure 5: Comparison of WER before and after correction for lower-performing ASR systems across datasets. The graph shows the WER distribution of NVIDIA Canary-1B (green), Azure STT (orange), and their LLM-corrected versions using Gemini 1.5 Pro (olive for NVIDIA correction, brown for Azure STT correction).