Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation

TL;DR

This work tackles the challenge of reliable multimodal medical image segmentation by integrating Dempster-Shafer theory (DST) with deep learning to quantify and fuse uncertainty. A deep evidential fusion framework is proposed, comprising feature-extraction modules, evidential-mapping modules, and a multimodality evidence fusion (MMEF) module that applies contextual discounting to modality-specific evidence before combining it with Dempster's rule. The training objective combines modality-level Dice losses with a fused-output Dice loss, enabling end-to-end optimization of feature extractors, evidence prototypes, and reliability coefficients for each modality/class. Empirical results on PET-CT lymphoma and BraTS2021 brain tumor datasets show improved segmentation accuracy and calibrated uncertainty, with learned reliability coefficients aligning with domain knowledge and offering interpretability for the fusion process. The method provides a scalable, transparent approach to uncertainty-aware multimodal fusion, with potential applicability beyond medical imaging to other heterogeneous data fusion tasks.

Abstract

Single-modality medical images generally do not contain enough information to reach an accurate and reliable diagnosis. For this reason, physicians generally diagnose diseases based on multimodal medical images such as, e.g., PET/CT. The effective fusion of multimodal information is essential to reach a reliable decision and explain how the decision is made as well. In this paper, we propose a fusion framework for multimodal medical image segmentation based on deep learning and the Dempster-Shafer theory of evidence. In this framework, the reliability of each single modality image when segmenting different objects is taken into account by a contextual discounting operation. The discounted pieces of evidence from each modality are then combined by Dempster's rule to reach a final decision. Experimental results with a PET-CT dataset with lymphomas and a multi-MRI dataset with brain tumors show that our method outperforms the state-of-the-art methods in accuracy and reliability.

Figures (10)

• Figure 1: The evidential neural network model.
• Figure 2: The proposed deep evidential fusion framework. It is composed of encoder-decoder feature extraction (FE) modules that represent images using deep features, evidence mapping (EM) modules that map deep features into mass functions, and a multimodal evidence fusion (MMEF) module that combines evidence from different modalities.
• Figure 3: Schematic description of a UNet-based FE module. The network consists of a contracting path (down-sampling layers) and an expansive path (up-sampling layers), which gives it the u-shaped architecture. Reproduced based on kerfoot2018left.
• Figure 4: Example of a patient with lymphomas. The first and second rows showcase, respectively, CT and PET slices, depicting axial, sagittal, and coronal views. The lymphomas correspond to the bright regions in PET slices.
• Figure 5: Examples of a patient with brain tumors in four MRI modalities: FLAIR, T1Gd, T1, and T2. The first and second rows show, respectively, the original images and the images with tumor masks for the three classes: peritumoral edema (ED, green), enhancing tumor (ET, yellow), and necrotic tumor core or non-enhancing tumor (NCR/NET, red).

Theorems & Definitions (2)

• Example 1
• Remark 1