3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation

TL;DR

This work addresses the poor performance of the 2D SAM on 3D medical tumor segmentation by introducing 3DSAM-adapter, a holistic, parameter-efficient 2D-to-3D adaptation. It modifies the image encoder to process volumetric inputs, replaces the prompt encoding with a visual sampler, and redesigns the mask decoder into a lightweight 3D architecture with multi-layer aggregation, while freezing most pre-trained weights and training only lightweight adapters. Across four public tumor datasets, the method achieves state-of-the-art results with a single volume-level prompt on three tasks and demonstrates robustness to prompt variations and minimal additional parameters. The approach offers a practical, scalable path to repurposing foundation models for domain-specific 3D medical image analysis.

Abstract

Despite that the segment anything model (SAM) achieved impressive results on general-purpose semantic segmentation with strong generalization ability on daily images, its demonstrated performance on medical image segmentation is less precise and not stable, especially when dealing with tumor segmentation tasks that involve objects of small sizes, irregular shapes, and low contrast. Notably, the original SAM architecture is designed for 2D natural images, therefore would not be able to extract the 3D spatial information from volumetric medical data effectively. In this paper, we propose a novel adaptation method for transferring SAM from 2D to 3D for promptable medical image segmentation. Through a holistically designed scheme for architecture modification, we transfer the SAM to support volumetric inputs while retaining the majority of its pre-trained parameters for reuse. The fine-tuning process is conducted in a parameter-efficient manner, wherein most of the pre-trained parameters remain frozen, and only a few lightweight spatial adapters are introduced and tuned. Regardless of the domain gap between natural and medical data and the disparity in the spatial arrangement between 2D and 3D, the transformer trained on natural images can effectively capture the spatial patterns present in volumetric medical images with only lightweight adaptations. We conduct experiments on four open-source tumor segmentation datasets, and with a single click prompt, our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation. We also compare our adaptation method with existing popular adapters, and observed significant performance improvement on most datasets.

Figures (7)

• Figure 1: Overview of our proposed method for 3DSAM-adapter. The original ViT is modified to support volumetric inputs. The prompt encoder is redesigned to support 3D point prompt, and the mask decoder is updated to 3D CNN with multi-layer aggregation to generate 3D segmentation.
• Figure 2: Spatial adapter.
• Figure 3: Structure of our prompt encoder based on visual sampler and global queries cross-attention.
• Figure 4: Structure of our lightweight mask decoder with multi-layer aggregation.
• Figure 5: Qualitative visualizations of the proposed method and baseline approaches on kidney tumor, pancreas tumor, liver tumor and colon cancer segmentation tasks.