Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sub-Region-Aware Modality Fusion and Adaptive Prompting for Multi-Modal Brain Tumor Segmentation

Shadi Alijani, Fereshteh Aghaee Meibodi, Homayoun Najjaran

TL;DR

The paper tackles the challenge of adapting foundation models to multi-modal medical imaging for brain tumor segmentation by introducing SOFA, a framework that combines sub-region-aware modality attention with adaptive prompt engineering to perform sub-region-specific fusion and iterative refinement. Built on a LiteMedSAM backbone, it processes 3D BraTS data with a 4-modality MRI setup and a 3D-aware training scheme, achieving notable improvements, especially in the necrotic core, over both single-modality baselines and naive multi-modal fusion. Key contributions include the data preprocessing pipeline, the sub-region attention mechanism with learnable weights per tissue type, and an adaptive prompting pipeline that refines segmentation through region-specific prompts, as well as comprehensive ablations and sub-region analyses. The results demonstrate that proper adaptation of foundation-model prompts and modality fusion significantly enhances segmentation accuracy and robustness, suggesting a practical path toward more reliable multi-modal medical-image solutions across institutions and modalities.

Abstract

The successful adaptation of foundation models to multi-modal medical imaging is a critical yet unresolved challenge. Existing models often struggle to effectively fuse information from multiple sources and adapt to the heterogeneous nature of pathological tissues. To address this, we introduce a novel framework for adapting foundation models to multi-modal medical imaging, featuring two key technical innovations: sub-region-aware modality attention and adaptive prompt engineering. The attention mechanism enables the model to learn the optimal combination of modalities for each tumor sub-region, while the adaptive prompting strategy leverages the inherent capabilities of foundation models to refine segmentation accuracy. We validate our framework on the BraTS 2020 brain tumor segmentation dataset, demonstrating that our approach significantly outperforms baseline methods, particularly in the challenging necrotic core sub-region. Our work provides a principled and effective approach to multi-modal fusion and prompting, paving the way for more accurate and robust foundation model-based solutions in medical imaging.

Sub-Region-Aware Modality Fusion and Adaptive Prompting for Multi-Modal Brain Tumor Segmentation

TL;DR

The paper tackles the challenge of adapting foundation models to multi-modal medical imaging for brain tumor segmentation by introducing SOFA, a framework that combines sub-region-aware modality attention with adaptive prompt engineering to perform sub-region-specific fusion and iterative refinement. Built on a LiteMedSAM backbone, it processes 3D BraTS data with a 4-modality MRI setup and a 3D-aware training scheme, achieving notable improvements, especially in the necrotic core, over both single-modality baselines and naive multi-modal fusion. Key contributions include the data preprocessing pipeline, the sub-region attention mechanism with learnable weights per tissue type, and an adaptive prompting pipeline that refines segmentation through region-specific prompts, as well as comprehensive ablations and sub-region analyses. The results demonstrate that proper adaptation of foundation-model prompts and modality fusion significantly enhances segmentation accuracy and robustness, suggesting a practical path toward more reliable multi-modal medical-image solutions across institutions and modalities.

Abstract

The successful adaptation of foundation models to multi-modal medical imaging is a critical yet unresolved challenge. Existing models often struggle to effectively fuse information from multiple sources and adapt to the heterogeneous nature of pathological tissues. To address this, we introduce a novel framework for adapting foundation models to multi-modal medical imaging, featuring two key technical innovations: sub-region-aware modality attention and adaptive prompt engineering. The attention mechanism enables the model to learn the optimal combination of modalities for each tumor sub-region, while the adaptive prompting strategy leverages the inherent capabilities of foundation models to refine segmentation accuracy. We validate our framework on the BraTS 2020 brain tumor segmentation dataset, demonstrating that our approach significantly outperforms baseline methods, particularly in the challenging necrotic core sub-region. Our work provides a principled and effective approach to multi-modal fusion and prompting, paving the way for more accurate and robust foundation model-based solutions in medical imaging.
Paper Structure (19 sections, 7 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 7 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation Models and Deep Learning in Medical Image Segmentation
  4. 3D and Volumetric Medical Image Segmentation
  5. Multi-Modal Fusion Strategies
  6. Challenges in Multi-Modal 3D Medical Image Segmentation
  7. Method
  8. Problem Formulation
  9. Data Preprocessing Pipeline
  10. Sub-Region-Aware Modality Attention and Adaptive Prompt Engineering
  11. Training Procedure
  12. Experiments
  13. Dataset and Evaluation Setup
  14. Single vs. Multi-Modal Performance
  15. Ablation Study: Framework Performance with Attention and Prompting
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An overview of the proposed framework, illustrating the sub-region-aware modality attention and adaptive prompt engineering.
  • Figure 2: Performance comparison with state-of-the-art methods on BraTS 2020. Our method achieves competitive whole tumor (WT) performance (0.900 vs. 0.890 for nnU-Net), and significantly outperforms baselines on enhancing tumor (ET) segmentation (0.900 vs. 0.820 for nnU-Net, +9.8% improvement).
  • Figure 3: Qualitative 3D visualization of sub-region-specific segmentation results from BraTS 2020. The subfigures show necrotic core (red), enhancing tumor (orange), and edema (purple) with ground truth (solid) and predictions (overlays).