Exploring Deeper! Segment Anything Model with Depth Perception for Camouflaged Object Detection

TL;DR

This work tackles the limitations of the Segment Anything Model (SAM) for Camouflaged Object Detection by introducing DSAM, a SAM-based RGB-D COD framework. DSAM comprises two key modules: the Prompt-Deeper Module (PDM), which uses knowledge distillation to fuse depth and RGB information into a depth-aware box prompt, and the Finer Module (FM), which refines SAM outputs by exploring missed regions through mask reversion and depth-guided attention. The model keeps SAM’s image and prompt encoders largely fixed while training the mask decoder, and it combines RGB-D cues to achieve robust COD performance with reduced training resources. Empirically, DSAM demonstrates competitive or state-of-the-art results across COD benchmarks, including CAMO, COD10K, and NC4K, and against RGB-D SOD methods with source-free depth, highlighting the practical impact of integrating depth perception into a powerful visual foundation model. The work also discusses an explicit combination of depth-driven refinement and RGB-D prompts, offering a scalable path for multimodal COD in challenging camouflage scenarios.

Abstract

This paper introduces a new Segment Anything Model with Depth Perception (DSAM) for Camouflaged Object Detection (COD). DSAM exploits the zero-shot capability of SAM to realize precise segmentation in the RGB-D domain. It consists of the Prompt-Deeper Module and the Finer Module. The Prompt-Deeper Module utilizes knowledge distillation and the Bias Correction Module to achieve the interaction between RGB features and depth features, especially using depth features to correct erroneous parts in RGB features. Then, the interacted features are combined with the box prompt in SAM to create a prompt with depth perception. The Finer Module explores the possibility of accurately segmenting highly camouflaged targets from a depth perspective. It uncovers depth cues in areas missed by SAM through mask reversion, self-filtering, and self-attention operations, compensating for its defects in the COD domain. DSAM represents the first step towards the SAM-based RGB-D COD model. It maximizes the utilization of depth features while synergizing with RGB features to achieve multimodal complementarity, thereby overcoming the segmentation limitations of SAM and improving its accuracy in COD. Experimental results on COD benchmarks demonstrate that DSAM achieves excellent segmentation performance and reaches the state-of-the-art (SOTA) on COD benchmarks with less consumption of training resources. The code will be available at https://github.com/guobaoxiao/DSAM.

Figures (5)

• Figure 1: Overview of our DSAM framework. It mainly includes three parts: SAM, the Prompt Deeper Module (PDM), and the Finer Module (FM). PDM consists of the Bias Correction Module (BCM), the Prompt Fuse Module (PFM) and Discrete Wavelet Transform (DWT). Regarding modules in SAM and PVT, the parameters in the module with snowflake are fixed, while the parameters in the module with spark can be optimized by training. In order to enhance the comprehensibility of the figure, we annotate the sources of the inputs to FM with arrows. Purple arrows indicate that $Pred_{SAM}$ originates from the output of SAM. Blue arrows represent depth embedding originating from the intermediate embedding of the PDM.
• Figure 2: Structure diagram of Bias Correction Module and Prompt Fuse Module. $CP$ represents channel projection, including $Up\ conv$ and $down\ conv$. $Up\ conv$ refers to convolution that increase the number of channels, while $down\ conv$ refers to convolution that decrease the number of channels. $DCs$ denotes the composition of convolution.
• Figure 3: Comparison of our DSAM and other methods in COD. Our approach enables a more comprehensive segmentation of camouflaged targets (row four and eight), while also exhibiting precise attention to detail (row one and three). DSAM achieves relatively satisfactory segmentation results in multi-objective scenarios as well (row nine to eleven). Better to zoom in.
• Figure 4: Ablation experiments on the input to the FM. $\uparrow$ indicates the higher the score the better and $\downarrow$ indicates the lower the score the better. $I$ stands for image embedding acting as student embedding, and $D$ stands for depth embedding acting as teacher embedding.
• Figure 5: Ablation experiments on the layer number to the FM. $Avg\_x$ represents the average score of positive indicators when the number of layers is $x$. $Mae\_x$ represents the $Mae$ when the number of layers is $x$.