Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image

TL;DR

This work tackles 3D multi-modal tumor segmentation by addressing the limitations of CNNs in local feature learning and Transformers in computational cost. It introduces a Mamba-based framework with a specific-modality encoder, an adaptive bi-level synergistic integration block for cross-modal fusion, and a decoder that preserves spatial detail. Ablation and cross-dataset experiments on BraTS2023 (MRI) and Hecktor2022 (PET/CT) demonstrate competitive or superior performance relative to CNN, Transformer, and prior Mamba-based methods, validating the effectiveness of modality-specific feature extraction and dynamic fusion. The proposed approach offers a scalable, flexible paradigm for leveraging modality-specific information, potentially improving clinical decision support in brain and head-and-neck tumor segmentation.

Abstract

Multi-modal 3D medical image segmentation aims to accurately identify tumor regions across different modalities, facing challenges from variations in image intensity and tumor morphology. Traditional convolutional neural network (CNN)-based methods struggle with capturing global features, while Transformers-based methods, despite effectively capturing global context, encounter high computational costs in 3D medical image segmentation. The Mamba model combines linear scalability with long-distance modeling, making it a promising approach for visual representation learning. However, Mamba-based 3D multi-modal segmentation still struggles to leverage modality-specific features and fuse complementary information effectively. In this paper, we propose a Mamba based feature extraction and adaptive multilevel feature fusion for 3D tumor segmentation using multi-modal medical image. We first develop the specific modality Mamba encoder to efficiently extract long-range relevant features that represent anatomical and pathological structures present in each modality. Moreover, we design an bi-level synergistic integration block that dynamically merges multi-modal and multi-level complementary features by the modality attention and channel attention learning. Lastly, the decoder combines deep semantic information with fine-grained details to generate the tumor segmentation map. Experimental results on medical image datasets (PET/CT and MRI multi-sequence) show that our approach achieve competitive performance compared to the state-of-the-art CNN, Transformer, and Mamba-based approaches.

Figures (3)

• Figure 1: Our architecture mainly includes Mamba block, bi-level synergistic integration block, and Res block, shown through a brain tumor segmentation with 4-sequence MRI.
• Figure 2: Qualitative results on BraTS2023 dataset.
• Figure 3: Qualitative results on Hecktor2022 dataset.