Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hybrid Mamba-SAM Architecture for Efficient 3D Medical Image Segmentation

Mohammadreza Gholipour Shahraki, Mehdi Rezaeian, Mohammad Ghasemzadeh

TL;DR

The paper tackles the problem of adapting a powerful foundation model (SAM) to 3D medical image segmentation, where domain shift, 2D limitations, and costly fine-tuning hinder clinical applicability. It proposes Mamba-SAM, a hybrid framework that freezes SAM's encoder and integrates efficient Mamba-based state-space modules through two strategies: a Dual-Branch MambaSAM that fuses frozen SAM features with a trainable specialist via cross-attention, and a 3D Adapter TP-Mamba-SAM that injects Tri-Plane Mamba adapters (with optional LoRA) into the SAM backbone to capture volumetric context, including a Multi-Frequency Gated Convolution (MFGC) variant for frequency-aware feature representation. On the ACDC cardiac MRI dataset, the Dual-Branch MambaSAM-Base achieves a mean Dice of $0.906$, competitive with UNet++ and superior on Myo ($0.910$) and LV ($0.971$), while the adapter-based TP_MFGC variant delivers fast inference at $4.77$ FPS with Dice $0.880$, and the TP-Mamba (LoRA) variant demonstrates outstanding parameter efficiency with very low VRAM usage. Overall, the study shows that hybridizing frozen foundation-model encoders with efficient, PEFT-enabled Mamba architectures can deliver high-accuracy 3D segmentation with favorable speed and memory profiles, enabling practical deployment in resource-constrained clinical settings.

Abstract

Accurate segmentation of 3D medical images such as MRI and CT is essential for clinical diagnosis and treatment planning. Foundation models like the Segment Anything Model (SAM) provide powerful general-purpose representations but struggle in medical imaging due to domain shift, their inherently 2D design, and the high computational cost of fine-tuning. To address these challenges, we propose Mamba-SAM, a novel and efficient hybrid architecture that combines a frozen SAM encoder with the linear-time efficiency and long-range modeling capabilities of Mamba-based State Space Models (SSMs). We investigate two parameter-efficient adaptation strategies. The first is a dual-branch architecture that explicitly fuses general features from a frozen SAM encoder with domain-specific representations learned by a trainable VMamba encoder using cross-attention. The second is an adapter-based approach that injects lightweight, 3D-aware Tri-Plane Mamba (TPMamba) modules into the frozen SAM ViT encoder to implicitly model volumetric context. Within this framework, we introduce Multi-Frequency Gated Convolution (MFGC), which enhances feature representation by jointly analyzing spatial and frequency-domain information via 3D discrete cosine transforms and adaptive gating. Extensive experiments on the ACDC cardiac MRI dataset demonstrate the effectiveness of the proposed methods. The dual-branch Mamba-SAM-Base model achieves a mean Dice score of 0.906, comparable to UNet++ (0.907), while outperforming all baselines on Myocardium (0.910) and Left Ventricle (0.971) segmentation. The adapter-based TP MFGC variant offers superior inference speed (4.77 FPS) with strong accuracy (0.880 Dice). These results show that hybridizing foundation models with efficient SSM-based architectures provides a practical and effective solution for 3D medical image segmentation.

A Hybrid Mamba-SAM Architecture for Efficient 3D Medical Image Segmentation

TL;DR

The paper tackles the problem of adapting a powerful foundation model (SAM) to 3D medical image segmentation, where domain shift, 2D limitations, and costly fine-tuning hinder clinical applicability. It proposes Mamba-SAM, a hybrid framework that freezes SAM's encoder and integrates efficient Mamba-based state-space modules through two strategies: a Dual-Branch MambaSAM that fuses frozen SAM features with a trainable specialist via cross-attention, and a 3D Adapter TP-Mamba-SAM that injects Tri-Plane Mamba adapters (with optional LoRA) into the SAM backbone to capture volumetric context, including a Multi-Frequency Gated Convolution (MFGC) variant for frequency-aware feature representation. On the ACDC cardiac MRI dataset, the Dual-Branch MambaSAM-Base achieves a mean Dice of , competitive with UNet++ and superior on Myo () and LV (), while the adapter-based TP_MFGC variant delivers fast inference at FPS with Dice , and the TP-Mamba (LoRA) variant demonstrates outstanding parameter efficiency with very low VRAM usage. Overall, the study shows that hybridizing frozen foundation-model encoders with efficient, PEFT-enabled Mamba architectures can deliver high-accuracy 3D segmentation with favorable speed and memory profiles, enabling practical deployment in resource-constrained clinical settings.

Abstract

Accurate segmentation of 3D medical images such as MRI and CT is essential for clinical diagnosis and treatment planning. Foundation models like the Segment Anything Model (SAM) provide powerful general-purpose representations but struggle in medical imaging due to domain shift, their inherently 2D design, and the high computational cost of fine-tuning. To address these challenges, we propose Mamba-SAM, a novel and efficient hybrid architecture that combines a frozen SAM encoder with the linear-time efficiency and long-range modeling capabilities of Mamba-based State Space Models (SSMs). We investigate two parameter-efficient adaptation strategies. The first is a dual-branch architecture that explicitly fuses general features from a frozen SAM encoder with domain-specific representations learned by a trainable VMamba encoder using cross-attention. The second is an adapter-based approach that injects lightweight, 3D-aware Tri-Plane Mamba (TPMamba) modules into the frozen SAM ViT encoder to implicitly model volumetric context. Within this framework, we introduce Multi-Frequency Gated Convolution (MFGC), which enhances feature representation by jointly analyzing spatial and frequency-domain information via 3D discrete cosine transforms and adaptive gating. Extensive experiments on the ACDC cardiac MRI dataset demonstrate the effectiveness of the proposed methods. The dual-branch Mamba-SAM-Base model achieves a mean Dice score of 0.906, comparable to UNet++ (0.907), while outperforming all baselines on Myocardium (0.910) and Left Ventricle (0.971) segmentation. The adapter-based TP MFGC variant offers superior inference speed (4.77 FPS) with strong accuracy (0.880 Dice). These results show that hybridizing foundation models with efficient SSM-based architectures provides a practical and effective solution for 3D medical image segmentation.
Paper Structure (18 sections, 6 equations, 3 figures, 2 tables)

This paper contains 18 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CNN-based Segmentation: The U-Net Era
  4. Transformers in Medical Vision
  5. Foundation Models and SAM Adaptation
  6. State Space Models (Mamba) in Vision
  7. Materials and Methods
  8. Architecture 1: Dual-Branch MambaSAM
  9. Architecture 2: 3D Adapter TP-Mamba-SAM
  10. Dataset and Preprocessing
  11. Training Details
  12. Results
  13. Evaluation Metrics
  14. Quantitative Results Analysis
  15. Performance, Efficiency, and Resource Usage
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Detailed schematic of the Dual-Branch MambaSAM, highlighting the flow of information, feature dimensions, and trainable components.
  • Figure 2: Detailed illustration of how the TP-Mamba adapter (potentially enhanced with MFGC replacing the CNN path) is inserted into a frozen SAM ViT block and its internal mechanism for processing 3D context.
  • Figure 3: Qualitative segmentation results for the TP_MFGC model on two different test slices. Red: RV, Green: Myo, Blue: LV. The predictions demonstrate high fidelity to the ground truth boundaries, potentially aided by the MFGC component's frequency analysis.