SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation

TL;DR

This work tackles the reliance of SAM on manual prompts and the absence of depth cues in RGB-only instance segmentation. It introduces SPDA-SAM, a depth-aware architecture with a dual-path encoder and a self-prompted decoder, featuring a Semantic-Spatial Self-prompt Module (SSSPM) and a Coarse-to-Fine RGB-D Fusion Module (C2FFM). The method eliminates human prompts and integrates monocular depth information to guide global structure and local details, achieving state-of-the-art results across 12 diverse datasets with favorable efficiency. The findings demonstrate that self-prompt guidance and depth fusion substantially boost segmentation accuracy, enhancing SAM’s applicability to depth-rich, domain-diverse scenarios.

Abstract

Recently, Segment Anything Model (SAM) has demonstrated strong generalizability in various instance segmentation tasks. However, its performance is severely dependent on the quality of manual prompts. In addition, the RGB images that instance segmentation methods normally use inherently lack depth information. As a result, the ability of these methods to perceive spatial structures and delineate object boundaries is hindered. To address these challenges, we propose a Self-prompted Depth-Aware SAM (SPDA-SAM) for instance segmentation. Specifically, we design a Semantic-Spatial Self-prompt Module (SSSPM) which extracts the semantic and spatial prompts from the image encoder and the mask decoder of SAM, respectively. Furthermore, we introduce a Coarse-to-Fine RGB-D Fusion Module (C2FFM), in which the features extracted from a monocular RGB image and the depth map estimated from it are fused. In particular, the structural information in the depth map is used to provide coarse-grained guidance to feature fusion, while local variations in depth are encoded in order to fuse fine-grained feature representations. To our knowledge, SAM has not been explored in such self-prompted and depth-aware manners. Experimental results demonstrate that our SPDA-SAM outperforms its state-of-the-art counterparts across twelve different data sets. These promising results should be due to the guidance of the self-prompts and the compensation for the spatial information loss by the coarse-to-fine RGB-D fusion operation.

Figures (7)

• Figure 1: Comparison between five state-of-the-art methods and our SPDA-SAM across 12 instance segmentation data sets.
• Figure 2: The architecture of the Self-prompted Depth-Aware SAM (SPDA-SAM), which consists of a depth-aware dual-path encoder and a self-prompted decoder. The encoder contains an RGB encoding path, a depth encoding path and a Coarse-to-Fine RGB-D Fusion Module (C2FFM). The decoder includes a mask decoder and a Semantic-Spatial Self-prompt Module (SSSPM).
• Figure 3: Visual comparison of the depth maps obtained using four depth estimators from four underwater hull images in the LIACI waszak2022semantic data set.
• Figure 4: Visualization of the results derived using two state-of-the-art baselines, i.e., RSPrompter chen2024rsprompter and USIS-SAM lian2024diving, and our method on the 10 images in the COME15K-E zhang2021rgb data set.
• Figure 5: Visualization of the results derived using two state-of-the-art baselines, i.e., RSPrompter chen2024rsprompter and USIS-SAM lian2024diving, and our method on the 10 images in the USIS10K lian2024diving data set.