Fuse4Seg: Image-Level Fusion Based Multi-Modality Medical Image Segmentation

TL;DR

A novel image-level fusion based multi-modality medical image segmentation method, Fuse4Seg, which is a bi-level learning framework designed to model the intertwined dependencies between medical image segmentation and medical image fusion.

Abstract

Although multi-modality medical image segmentation holds significant potential for enhancing the diagnosis and understanding of complex diseases by integrating diverse imaging modalities, existing methods predominantly rely on feature-level fusion strategies. We argue the current feature-level fusion strategy is prone to semantic inconsistencies and misalignments across various imaging modalities because it merges features at intermediate layers in a neural network without evaluative control. To mitigate this, we introduce a novel image-level fusion based multi-modality medical image segmentation method, Fuse4Seg, which is a bi-level learning framework designed to model the intertwined dependencies between medical image segmentation and medical image fusion. The image-level fusion process is seamlessly employed to guide and enhance the segmentation results through a layered optimization approach. Besides, the knowledge gained from the segmentation module can effectively enhance the fusion module. This ensures that the resultant fused image is a coherent representation that accurately amalgamates information from all modalities. Moreover, we construct a BraTS-Fuse benchmark based on BraTS dataset, which includes 2040 paired original images, multi-modal fusion images, and ground truth. This benchmark not only serves image-level medical segmentation but is also the largest dataset for medical image fusion to date. Extensive experiments on several public datasets and our benchmark demonstrate the superiority of our approach over prior state-of-the-art (SOTA) methodologies.

Figures (6)

• Figure 1: (left) The bi-level optimization learning process with fusion task as the leader and segmentation task as follower. (right) Existing Multi-Modality Medical Image Segmentation Methods vs. Our Fuse4Seg. The DHE and DLE are donotes to Deep High-frequency Encoder and Deep Low-frequency Encoder, respectively.
• Figure 2: The overall framework of our Fuse4Seg, which consist of a fusion module and a segmentation module.
• Figure 3: The diagram illustrates the architecture of a pre-trained dual-stream decomposition encoder.
• Figure 4: The data processing pipeline of our BraTS-Fuse dataset and samples of both source images and fused images. As a reference, our dataset, which built upon the BraTS2021 dataset, contains 2,040 aligned cases.
• Figure 5: The fusion and segmentation results.