MERA: Multimodal and Multiscale Self-Explanatory Model with Considerably Reduced Annotation for Lung Nodule Diagnosis

TL;DR

MERA tackles the challenge of accurate lung nodule diagnosis under limited annotations by combining unsupervised feature learning with weakly supervised, multimodal explanations. It employs a Vision Transformer backbone trained with self-supervised contrastive learning (Stage 1) and a sparse, semi-supervised, active-learning-driven Stage 2 to predict malignancy and extract human-interpretable nodule attributes. The model delivers model-level global explanations via latent-space semantic clustering, instance-level case-based explanations, local visual attention maps, and concept explanations with minimal annotation (as little as 1%), achieving competitive or superior performance on the LIDC dataset while reducing labeling costs. These multimodal explanations are integrated into the decision process rather than added post hoc, improving transparency and trust in clinical settings, and the approach is validated through extensive ablations and case studies. The work advances explainable AI for medical imaging by demonstrating robust, interpretable, and annotation-efficient lung cancer decision support, with open-source code for reproduction.

Abstract

Lung cancer, a leading cause of cancer-related deaths globally, emphasises the importance of early detection for better patient outcomes. Pulmonary nodules, often early indicators of lung cancer, necessitate accurate, timely diagnosis. Despite Explainable Artificial Intelligence (XAI) advances, many existing systems struggle providing clear, comprehensive explanations, especially with limited labelled data. This study introduces MERA, a Multimodal and Multiscale self-Explanatory model designed for lung nodule diagnosis with considerably Reduced Annotation requirements. MERA integrates unsupervised and weakly supervised learning strategies (self-supervised learning techniques and Vision Transformer architecture for unsupervised feature extraction) and a hierarchical prediction mechanism leveraging sparse annotations via semi-supervised active learning in the learned latent space. MERA explains its decisions on multiple levels: model-level global explanations via semantic latent space clustering, instance-level case-based explanations showing similar instances, local visual explanations via attention maps, and concept explanations using critical nodule attributes. Evaluations on the public LIDC dataset show MERA's superior diagnostic accuracy and self-explainability. With only 1% annotated samples, MERA achieves diagnostic accuracy comparable to or exceeding state-of-the-art methods requiring full annotation. The model's inherent design delivers comprehensive, robust, multilevel explanations aligned closely with clinical practice, enhancing trustworthiness and transparency. Demonstrated viability of unsupervised and weakly supervised learning lowers the barrier to deploying diagnostic AI in broader medical domains. Our complete code is open-source available: https://github.com/diku-dk/credanno.

Figures (19)

• Figure 1: Multimodal and multiscale explanations as an intrinsic driving force of the decision: local visual explanations via attention maps, model-level global explanations through semantic latent space clustering, instance-level case-based explanations providing similar instances, and concept explanations based on critical nodule attributes.
• Figure 2: Method overview of the data-/annotation-efficient training. In Stage 1, an encoder is trained using self-supervised contrastive learning to map the input nodule images to a semantically meaningful latent space. In Stage 2, the proposed annotation exploitation mechanism conducts semi-supervised active learning with sparse seeding and training quenching in the learned space, to jointly exploit the extracted features, annotations, and unlabelled data.
• Figure 3: t-SNE visualisation of features extracted from testing images. Data points are coloured using ground truth annotations. Malignancy shows highly separable in the learned space, and semantically correlates with the clustering in each nodule attribute.
• Figure 4: Case-based explanation of a malignant nodule. Nearest training and testing samples illustrating similarities in poorly defined margins, lobulation, and spiculation, which are typical morphological characteristics suggesting malignancy.
• Figure 5: Case-based explanation of a benign nodule.. Nearest training and testing samples display the same solid texture, smooth and sharp margins, and lack of lobulations or spiculations, mirroring the features of the target nodule.