SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging
Tan-Hanh Pham, Chris Ngo, Trong-Duong Bui, Minh Luu Quang, Tan-Huong Pham, Truong-Son Hy
TL;DR
SilVar-Med addresses the need for hands-free, interpretable medical AI by introducing an end-to-end speech-driven Visual Language Model capable of reasoning-based abnormality detection in medical imaging. The approach integrates a speech encoder, PubMedCLIP visual encoder, and a reasoning-capable language model to produce structured explanations for radiologic abnormalities, complemented by a dedicated reasoning dataset and an LLM-as-Judge evaluation framework. Empirical results show that SilVar-Med achieves competitive performance against text-based baselines and outperforms cascaded speech-to-text pipelines on key metrics, while revealing both the promise and challenges of automated medical reasoning from spoken queries. Together with a two-stage training strategy and a reasoning-focused dataset, this work advances transparent, interactive clinical AI and highlights ongoing needs for speech-driven medical benchmarks and expert-aligned evaluation.
Abstract
Medical Visual Language Models have shown great potential in various healthcare applications, including medical image captioning and diagnostic assistance. However, most existing models rely on text-based instructions, limiting their usability in real-world clinical environments especially in scenarios such as surgery, text-based interaction is often impractical for physicians. In addition, current medical image analysis models typically lack comprehensive reasoning behind their predictions, which reduces their reliability for clinical decision-making. Given that medical diagnosis errors can have life-changing consequences, there is a critical need for interpretable and rational medical assistance. To address these challenges, we introduce an end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image assistant that integrates speech interaction with VLMs, pioneering the task of voice-based communication for medical image analysis. In addition, we focus on the interpretation of the reasoning behind each prediction of medical abnormalities with a proposed reasoning dataset. Through extensive experiments, we demonstrate a proof-of-concept study for reasoning-driven medical image interpretation with end-to-end speech interaction. We believe this work will advance the field of medical AI by fostering more transparent, interactive, and clinically viable diagnostic support systems. Our code and dataset are publicly available at SiVar-Med.