Transformer Architectures for Respiratory Sound Analysis and Multimodal Diagnosis

TL;DR

This work explores transformer-based architectures for asthma screening from respiratory sounds, introducing an Audio Spectrogram Transformer (AST) fine-tuned on medical data and a multimodal Vision-Language-like system that fuses spectrogram images with structured metadata. AST achieves approximately 97% accuracy (F1 ~0.97, ROC AUC ~0.98) for Asthma versus Not Asthma, significantly outperforming the internal DenseNet baseline. The multimodal VLM reaches about 86–87% accuracy, validating the integration of clinical context via prompts and adapters, and producing structured JSON outputs. Overall, the study demonstrates that self-attention captures non-local acoustic patterns effectively for screening while multimodal architectures enable context-aware, clinically interpretable inferences with practical deployment potential.

Abstract

Respiratory sound analysis is a crucial tool for screening asthma and other pulmonary pathologies, yet traditional auscultation remains subjective and experience-dependent. Our prior research established a CNN baseline using DenseNet201, which demonstrated high sensitivity in classifying respiratory sounds. In this work, we (i) adapt the Audio Spectrogram Transformer (AST) for respiratory sound analysis and (ii) evaluate a multimodal Vision-Language Model (VLM) that integrates spectrograms with structured patient metadata. AST is initialized from publicly available weights and fine-tuned on a medical dataset containing hundreds of recordings per diagnosis. The VLM experiment uses a compact Moondream-type model that processes spectrogram images alongside a structured text prompt (sex, age, recording site) to output a JSON-formatted diagnosis. Results indicate that AST achieves approximately 97% accuracy with an F1-score around 97% and ROC AUC of 0.98 for asthma detection, significantly outperforming both the internal CNN baseline and typical external benchmarks. The VLM reaches 86-87% accuracy, performing comparably to the CNN baseline while demonstrating the capability to integrate clinical context into the inference process. These results confirm the effectiveness of self-attention for acoustic screening and highlight the potential of multimodal architectures for holistic diagnostic tools.

Figures (4)

• Figure 1: Example TIFF spectrogram input
• Figure 2: VLM inference pipeline from spectrogram and prompt to JSON output
• Figure 3: Model performance comparison for Asthma vs Not Asthma
• Figure 4: Interpretability via visualization: Grad-CAM heatmap overlay on a spectrogram