Explainable Multi-Modal Deep Learning for Automatic Detection of Lung Diseases from Respiratory Audio Signals

TL;DR

The study tackles automatic multiclass lung-disease detection from respiratory sounds by proposing an explainable hybrid multimodal framework that fuses deep spectro-temporal representations (CNN–BiLSTM–Attention on mel-spectrograms) with handcrafted acoustic features. The model is trained and evaluated on the Asthma Detection Dataset Version 2, achieving 91.21% accuracy, a macro F1 of 0.899, and a macro ROC-AUC of 0.9866, with Grad-CAM, Integrated Gradients, and SHAP providing interpretable explanations aligned with known biomarkers. Ablation studies confirm the value of temporal modeling, attention, and multimodal fusion, while XAI analyses demonstrate clinically meaningful attributions across spectrogram regions and handcrafted descriptors. The framework offers strong potential for telemedicine and point-of-care respiratory screening, delivering both high diagnostic performance and transparent, biomarker-grounded reasoning.

Abstract

Respiratory diseases remain major global health challenges, and traditional auscultation is often limited by subjectivity, environmental noise, and inter-clinician variability. This study presents an explainable multimodal deep learning framework for automatic lung-disease detection using respiratory audio signals. The proposed system integrates two complementary representations: a spectral-temporal encoder based on a CNN-BiLSTM Attention architecture, and a handcrafted acoustic-feature encoder capturing physiologically meaningful descriptors such as MFCCs, spectral centroid, spectral bandwidth, and zero-crossing rate. These branches are combined through late-stage fusion to leverage both data-driven learning and domain-informed acoustic cues. The model is trained and evaluated on the Asthma Detection Dataset Version 2 using rigorous preprocessing, including resampling, normalization, noise filtering, data augmentation, and patient-level stratified partitioning. The study achieved strong generalization with 91.21% accuracy, 0.899 macro F1-score, and 0.9866 macro ROC-AUC, outperforming all ablated variants. An ablation study confirms the importance of temporal modeling, attention mechanisms, and multimodal fusion. The framework incorporates Grad-CAM, Integrated Gradients, and SHAP, generating interpretable spectral, temporal, and feature-level explanations aligned with known acoustic biomarkers to build clinical transparency. The findings demonstrate the framework's potential for telemedicine, point-of-care diagnostics, and real-world respiratory screening.

Figures (6)

• Figure 1: Proposed Hybrid Multimodal CNN–BiLSTM–Attention Framework with integrated XAI
• Figure 2: Comprehensive Data Preprocessing and Multimodal Feature Extraction Pipeline for Respiratory Sounds.
• Figure 3: Overview of the proposed hybrid multimodal CNN--BiLSTM--Attention architecture.
• Figure 4: Training and Validation Learning Curves for the Full Hybrid Model. (a) Evolution of accuracy on the training and validation sets. (b) Evolution of loss on the training and validation sets.
• Figure 5: Detailed Performance Analysis on the Independent Test Set: (a) One-vs-Rest Receiver Operating Characteristic (ROC) Curves for each respiratory class (b) Confusion Matrix showing the distribution of true versus predicted labels.