Towards Explainable Skin Cancer Classification: A Dual-Network Attention Model with Lesion Segmentation and Clinical Metadata Fusion

TL;DR

This work tackles the need for accurate and interpretable skin cancer classification from dermoscopic images. It introduces a Transformer Dual-Branch Network (TDBN) that combines a Deep-UNet segmentation backbone with dual DenseNet201 encoders and a multi-head cross-attention module, augmented by a transformer-based clinical metadata encoder. By fusing lesion-focused visual features and patient data, the approach achieves state-of-the-art segmentation on ISIC2018 and superior multi-class classification performance on HAM10000 and ISIC2019, with Grad-CAM heatmaps demonstrating lesion-centric explanations. The framework enhances clinical trust and applicability by demonstrating robust accuracy gains and explicit interpretability through attention mechanisms and heatmaps, validated on public benchmarks.

Abstract

Skin cancer is a life-threatening disease where early detection significantly improves patient outcomes. Automated diagnosis from dermoscopic images is challenging due to high intra-class variability and subtle inter-class differences. Many deep learning models operate as "black boxes," limiting clinical trust. In this work, we propose a dual-encoder attention-based framework that leverages both segmented lesions and clinical metadata to enhance skin lesion classification in terms of both accuracy and interpretability. A novel Deep-UNet architecture with Dual Attention Gates (DAG) and Atrous Spatial Pyramid Pooling (ASPP) is first employed to segment lesions. The classification stage uses two DenseNet201 encoders-one on the original image and another on the segmented lesion whose features are fused via multi-head cross-attention. This dual-input design guides the model to focus on salient pathological regions. In addition, a transformer-based module incorporates patient metadata (age, sex, lesion site) into the prediction. We evaluate our approach on the HAM10000 dataset and the ISIC 2018 and 2019 challenges. The proposed method achieves state-of-the-art segmentation performance and significantly improves classification accuracy and average AUC compared to baseline models. To validate our model's reliability, we use Gradient-weighted Class Activation Mapping (Grad-CAM) to generate heatmaps. These visualizations confirm that our model's predictions are based on the lesion area, unlike models that rely on spurious background features. These results demonstrate that integrating precise lesion segmentation and clinical data with attention-based fusion leads to a more accurate and interpretable skin cancer classification model.

Figures (7)

• Figure 1: Overview of the proposed Deep-UNet architecture for skin lesion segmentation. The model uses an EfficientNet-B3 encoder, an ASPP bottleneck for multi-scale feature extraction, and a decoder with Depthwise Attention Gated (DAG) skip connections and deep supervision outputs.
• Figure 2: Segmentation Process: Original image multiplied by binary mask to create the segmented image.
• Figure 3: Class distribution in the training dataset before (left) and after (right) balancing for the HAM10000 dataset.
• Figure 4: Class distribution in the training dataset before (left) and after (right) balancing for the ISIC 2019 dataset.
• Figure 5: Architecture of the proposed Transformer Dual-Branch Network (TDBN), integrating dual DenseNet encoders, multi-head cross-attention, transformer-based metadata embedding, and additive multimodal fusion.