SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation

TL;DR

SpeechMedAssist tackles the scarcity of medical speech data by introducing a two-stage training paradigm for SpeechLMs: first inject medical knowledge via large-scale medical text, then realign modalities with a limited amount of speech data. The authors construct TextMedDataset (405k samples) and SpeechMedDataset (198k samples) to support Stage I and II, and they establish SpeechMedBench to evaluate single-turn knowledge, multi-turn consultation skills, safety, and speech quality. Empirical results show that SpeechMedAssist achieves superior medical knowledge mastery, safety, and diagnostic communication while preserving general-domain knowledge and delivering high-quality, robust speech output. The framework demonstrates efficient adaptation of SpeechLMs to vertical domains with limited speech data, offering a practical blueprint for deploying speech-enabled medical dialogue systems at scale.

Abstract

Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and patient-unfriendly. Recent advances in speech language models (SpeechLMs) have enabled more natural speech-based interaction, yet the scarcity of medical speech data and the inefficiency of directly fine-tuning on speech data jointly hinder the adoption of SpeechLMs in medical consultation. In this paper, we propose SpeechMedAssist, a SpeechLM natively capable of conducting speech-based multi-turn interactions with patients. By exploiting the architectural properties of SpeechLMs, we decouple the conventional one-stage training into a two-stage paradigm consisting of (1) Knowledge & Capability Injection via Text and (2) Modality Re-alignment with Limited Speech Data, thereby reducing the requirement for medical speech data to only 10k synthesized samples. To evaluate SpeechLMs for medical consultation scenarios, we design a benchmark comprising both single-turn question answering and multi-turn simulated interactions. Experimental results show that our model outperforms all baselines in both effectiveness and robustness in most evaluation settings.

Figures (13)

• Figure 1: An illustration highlighting the limitations of text-based medical consultation, alongside the advantages of speech-based medical consultation.
• Figure 2: An overview of our work. Data Constraction: we construct TextMedDataset by filtering and rewriting collected medical text corpora, and build SpeechMedDataset by extracting patient information from dialogues and synthesizing matched speech. Model Architecture: we focus on the encoder–adaptor–LLM–decoder architecture, which supports text–speech dual-modal input and streaming output. Training Strategy: the first stage injects knowledge&capability into LLM core using TextMedDataset, while the second stage achieves modality re-alignment with a small amount of speech data from SpeechMedDataset.
• Figure 3: Comparison of our model with other models on multi-dimensions of multi-turn conversations metrics (a) MedDG and (b) AIHospital. Apart from a few dimensions that favor long-text responses, our model exhibits strong diagnostic capabilities.
• Figure 4: Win rates of our model against strong baselines, using Qwen2.5-72B and DeepSeek-V3.1-685B as patient simulators and GPT-4o as the judge. Our model achieves higher win rates in all settings.
• Figure 5: (a): Comparison of the performance between the model trained in Stage II and the model trained from scratch on speech data, for single-turn Q&A and multi-turn conversation evaluations across training steps. To ensure the reliability of our conclusions, we compute the variance at step 5k and 97k. (b): Comparison of conv score variations across training steps, where models are trained with different amounts of speech data. Remarkably, using only 10k audio samples yields performance close to that of a model trained with 198k samples.