Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer

TL;DR

This work tackles predicting pathological response ($pR$) to neoadjuvant therapy in NSCLC by introducing a Multimodal Doctor-in-the-Loop framework that fuses CT imaging and clinical data via intermediate fusion. The model integrates intrinsic explainability through Grad-CAM-guided loss terms and clinician-provided masks, with a training protocol that progressively focuses from global lung regions to lesion areas. Empirical results show that intermediate fusion with doctor-in-the-loop guidance yields superior predictive accuracy and explainability compared with unimodal approaches and other fusion schemes, highlighting the value of multimodal integration and clinically guided training. The approach offers a path toward non-invasive, robust, and interpretable pR prediction suitable for clinical decision support, and it lays the groundwork for validation on larger datasets and extension to other cancer endpoints.

Abstract

This study proposes a novel approach combining Multimodal Deep Learning with intrinsic eXplainable Artificial Intelligence techniques to predict pathological response in non-small cell lung cancer patients undergoing neoadjuvant therapy. Due to the limitations of existing radiomics and unimodal deep learning approaches, we introduce an intermediate fusion strategy that integrates imaging and clinical data, enabling efficient interaction between data modalities. The proposed Multimodal Doctor-in-the-Loop method further enhances clinical relevance by embedding clinicians' domain knowledge directly into the training process, guiding the model's focus gradually from broader lung regions to specific lesions. Results demonstrate improved predictive accuracy and explainability, providing insights into optimal data integration strategies for clinical applications.

Figures (3)

• Figure 1: Multimodal Doctor-in-the-Loop training paradigm: integration of clinical and imaging data. Step 0: Model trained on global image using classification loss ($L_{cls}$). Step 1: Segmentation mask $\boldsymbol{M}_{1}$ guides model focus via Grad-CAM heatmaps, training with classification ($L{cls}$) and XAI ($L_{xai}$) losses. Step 2: Detailed segmentation mask $\boldsymbol{M}_{2}$ further refines focus using classification ($L{cls}$) and XAI ($L_{xai}$) losses.
• Figure 2: Top row: Comparison between the unimodal models (CT and clinical) and the multimodal models trained via early, late and intermediate fusion, with intermediate fusion representing the proposed Multimodal Doctor-in-the-Loop. Bottom row: Comparison between Multimodal Doctor-in-the-Loop, Segmentation and XAI-guide approaches via early, late and intermediate fusion.
• Figure 3: Clinical Explainability: SHAP values of the 20 most significant clinical features, highlighting variables related to diagnosis, patient characteristics, and treatments. A color gradient is used to indicate the relevance of each feature (low: blue, high: red). Imaging Explainability: Grad-CAM heatmaps comparing unimodal and multimodal Doctor-in-the-Loop approaches for four representative patients categorized as No-pR or pR. For each patient, three visualizations are shown: the CT image with lesion segmentation, the corresponding 2D Grad-CAM map, and the corresponding 3D Grad-CAM map, with a color gradient indicating the heatmap intensity values (low: blue, high: red). The multimodal model demonstrates enhanced explainability through more refined focus.