Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechCT-CLIP: Distilling Text-Image Knowledge to Speech for Voice-Native Multimodal CT Analysis

Lukas Buess, Jan Geier, David Bani-Harouni, Chantal Pellegrini, Matthias Keicher, Paula Andrea Perez-Toro, Nassir Navab, Andreas Maier, Tomas Arias-Vergara

TL;DR

This paper tackles the gap between spoken radiology reports and multimodal CT analysis by proposing SpeechCT-CLIP, a vision–speech contrastive model that learns from a large-scale synthetic dataset Speech-RATE. A frozen CT-text CLIP teacher guides a trainable speech encoder through knowledge distillation, enabling robust speech-only inference for zero-shot abnormality classification and cross-modal case retrieval. The approach narrows the performance gap with text-based models (e.g., F1 improving from 0.623 to 0.705 and AUROC from 0.623 to 0.708 on internal data) and generalizes to external domains like RAD-ChestCT. These results demonstrate the feasibility and practical value of voice-native multimodal pretraining in clinical CT analysis, with potential extensions to multilingual and interactive AI workflows.

Abstract

Spoken communication plays a central role in clinical workflows. In radiology, for example, most reports are created through dictation. Yet, nearly all medical AI systems rely exclusively on written text. In this work, we address this gap by exploring the feasibility of learning visual-language representations directly from spoken radiology reports. Specifically, we synthesize a large-scale dataset (Speech-RATE) of spoken radiology reports and train SpeechCT-CLIP, a contrastive model that aligns speech and 3D CT volumes in a shared representation space. While naive speech-based models underperform compared to text-trained counterparts, we show that knowledge distillation from a pretrained text-image CLIP model effectively transfers semantic alignment capabilities from text to speech, substantially narrowing this gap. Experiments demonstrate improved zero-shot classification F1 from 0.623 to 0.705, recovering 88% of the performance difference, and strong retrieval results without requiring text at inference. These findings highlight speech as a practical alternative to text in multimodal pretraining and open the door to voice-driven diagnostic support tools in clinical practice.

SpeechCT-CLIP: Distilling Text-Image Knowledge to Speech for Voice-Native Multimodal CT Analysis

TL;DR

This paper tackles the gap between spoken radiology reports and multimodal CT analysis by proposing SpeechCT-CLIP, a vision–speech contrastive model that learns from a large-scale synthetic dataset Speech-RATE. A frozen CT-text CLIP teacher guides a trainable speech encoder through knowledge distillation, enabling robust speech-only inference for zero-shot abnormality classification and cross-modal case retrieval. The approach narrows the performance gap with text-based models (e.g., F1 improving from 0.623 to 0.705 and AUROC from 0.623 to 0.708 on internal data) and generalizes to external domains like RAD-ChestCT. These results demonstrate the feasibility and practical value of voice-native multimodal pretraining in clinical CT analysis, with potential extensions to multilingual and interactive AI workflows.

Abstract

Spoken communication plays a central role in clinical workflows. In radiology, for example, most reports are created through dictation. Yet, nearly all medical AI systems rely exclusively on written text. In this work, we address this gap by exploring the feasibility of learning visual-language representations directly from spoken radiology reports. Specifically, we synthesize a large-scale dataset (Speech-RATE) of spoken radiology reports and train SpeechCT-CLIP, a contrastive model that aligns speech and 3D CT volumes in a shared representation space. While naive speech-based models underperform compared to text-trained counterparts, we show that knowledge distillation from a pretrained text-image CLIP model effectively transfers semantic alignment capabilities from text to speech, substantially narrowing this gap. Experiments demonstrate improved zero-shot classification F1 from 0.623 to 0.705, recovering 88% of the performance difference, and strong retrieval results without requiring text at inference. These findings highlight speech as a practical alternative to text in multimodal pretraining and open the door to voice-driven diagnostic support tools in clinical practice.

Paper Structure

This paper contains 15 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Speech-RATE: Synthetic Spoken Report Dataset
  4. Contrastive Pretraining
  5. Knowledge Distillation
  6. Inference
  7. Experimental Setup
  8. Data Setup
  9. Evaluation
  10. Implementation Details
  11. Results and Discussion
  12. Speech Encoder Selection
  13. Zero-Shot Classification
  14. Case Retrieval
  15. Conclusion

Figures (2)

  • Figure 1: Overview of SpeechCT-CLIP. A frozen, pretrained CT and text encoder supervise the audio encoder. Spoken reports yield audio embeddings $\{A_i\}$, aligned with CT features $\{V_j\}$ via contrastive learning and distilled from text embeddings $\{T_i\}$.
  • Figure 2: (a) Zero-shot classification: spoken pathology queries are matched with CT features. (b) Case retrieval: spoken reports are used to retrieve corresponding CT volumes.