Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities

TL;DR

The paper addresses the challenge of brain tumor segmentation when MRI modalities are missing by introducing ACIDIS, a framework that combines Anatomical Consistency Distillation (ACD) and Modality Feature Synthesis Block (MFSB). ACID comprises ACCT and AFEB to transfer rich, structurally aligned anatomical information from multi-modal to mono-modal representations, while MFSB generates modality-specific features to compensate for missing data, with a joint loss L_Overall = L_ACCT + L_Seg + L_Syn. Key contributions include a variance- and covariance-based distillation mechanism that preserves anatomical structure, a feature synthesis pathway for missing modalities, and extensive validation on BraTS2018/2020 showing improved Dice scores over state-of-the-art methods. The approach has practical significance for clinical workflows where full multi-modal MRI is not always available, enabling more accurate and robust brain tumor segmentation.

Abstract

Multi-modal Magnetic Resonance Imaging (MRI) is imperative for accurate brain tumor segmentation, offering indispensable complementary information. Nonetheless, the absence of modalities poses significant challenges in achieving precise segmentation. Recognizing the shared anatomical structures between mono-modal and multi-modal representations, it is noteworthy that mono-modal images typically exhibit limited features in specific regions and tissues. In response to this, we present Anatomical Consistency Distillation and Inconsistency Synthesis (ACDIS), a novel framework designed to transfer anatomical structures from multi-modal to mono-modal representations and synthesize modality-specific features. ACDIS consists of two main components: Anatomical Consistency Distillation (ACD) and Modality Feature Synthesis Block (MFSB). ACD incorporates the Anatomical Feature Enhancement Block (AFEB), meticulously mining anatomical information. Simultaneously, Anatomical Consistency ConsTraints (ACCT) are employed to facilitate the consistent knowledge transfer, i.e., the richness of information and the similarity in anatomical structure, ensuring precise alignment of structural features across mono-modality and multi-modality. Complementarily, MFSB produces modality-specific features to rectify anatomical inconsistencies, thereby compensating for missing information in the segmented features. Through validation on the BraTS2018 and BraTS2020 datasets, ACDIS substantiates its efficacy in the segmentation of brain tumors with missing MRI modalities.

Figures (7)

• Figure 1: Different modalities share the same anatomical semantics, but they vary in terms of pixel intensity visualization. Typically, 1. The modality with richer information and clearer anatomical structure exhibits greater variance. 2. Greater anatomical similarity between modalities results in higher absolute covariance values.
• Figure 2: Overall architecture of our ACDIS. During the training phase, ACDIS comprises four mono-encoders for extracting mono-modal features, one auxiliary encoder dedicated to consistency distillation in conjunction with the proposed AFEB and ACCT, one auxiliary decoder for obtaining individual mono-modal segmentation (denoted in the yellow box), one MFSB designed to synthesize features for missing modalities, and one fusion decoder responsible for generating the final segmentation result. The auxiliary encoder and decoder are discarded during the inference phase.
• Figure 3: Anatomical Feature Enhancement Block (AFEB).
• Figure 4: Modality Feature Synthesis Block (MFSB).
• Figure 5: Segmentation results of different methods with various available modalities on BraTS2020.