Boosting Segment Anything Model to Generalize Visually Non-Salient Scenarios

TL;DR

This work tackles the challenge of segmenting visually non-salient objects where SAM's zero-shot performance degrades due to low foreground-background contrast. It introduces VNS-SAM, a lightweight extension built on frozen SAM that fuses a Mask-Edge Token Interactive (METI) decoder with a Non-Salient Feature Mining (NSFM) module to better capture subtle non-salient features. A unified VNS-SEG dataset comprising over 35K image-mask pairs enables robust training and comprehensive benchmarking across camouflaged, polyp, and low-light scenarios, with rigorous zero-shot evaluation on real-world unseen data. Experimental results show substantial improvements over SAM and domain-specific baselines while maintaining competitive generalization and efficiency, highlighting the approach's practicality for real-world VNS segmentation tasks.

Abstract

Segment Anything Model (SAM), known for its remarkable zero-shot segmentation capabilities, has garnered significant attention in the community. Nevertheless, its performance is challenged when dealing with what we refer to as visually non-salient scenarios, where there is low contrast between the foreground and background. In these cases, existing methods often cannot capture accurate contours and fail to produce promising segmentation results. In this paper, we propose Visually Non-Salient SAM (VNS-SAM), aiming to enhance SAM's perception of visually non-salient scenarios while preserving its original zero-shot generalizability. We achieve this by effectively exploiting SAM's low-level features through two designs: Mask-Edge Token Interactive decoder and Non-Salient Feature Mining module. These designs help the SAM decoder gain a deeper understanding of non-salient characteristics with only marginal parameter increments and computational requirements. The additional parameters of VNS-SAM can be optimized within 4 hours, demonstrating its feasibility and practicality. In terms of data, we established VNS-SEG, a unified dataset for various VNS scenarios, with more than 35K images, in contrast to previous single-task adaptations. It is designed to make the model learn more robust VNS features and comprehensively benchmark the model's segmentation performance and generalizability on VNS scenarios. Extensive experiments across various VNS segmentation tasks demonstrate the superior performance of VNS-SAM, particularly under zero-shot settings, highlighting its potential for broad real-world applications. Codes and datasets are publicly available at https://guangqian-guo.github.io/VNS-SAM.

Figures (9)

• Figure 1: A comparison of masks predicted by SAM and VNS-SAM under three typical non-salient scenarios. SAM (the second row) often struggles when dealing with (a) camouflaged objects where the objects perfectly match its surroundings, (b) polyp objects where polyp tissues and normal tissues have the same texture, posing challenges to medical image analysis, and (c) objects in low-light conditions where the targets lack significant color contrast with their backgrounds. SAM fails to accurately identify object boundaries and complete structures, leading to missing segmentation details and incorrect background predictions. In contrast, VNS-SAM (the third row) can produce more accurate segmentation. (Best viewed in color)
• Figure 2: (a) Finetuning the entire decoder of SAM, (b) Finetuning additional output mask token to predict refined mask, (c) Our VNS-SAM integrates the interaction of edge semantics and dual-level decoder layers enhancement. The informative mask and edge features in the encoder are extracted by the Non-salient Feature Mining (NSFM) module, enriching the representation of the prediction layers for accurate segmentation. (Best viewed in color)
• Figure 3: Overview. Building upon SAM, VNS-SAM makes two key improvements. First, it enhances the SAM's original decoder to a mask-edge token interactive (METI) decoder with the interaction of edge semantics and dual-level decoder layers enhancement. Second, a lightweight non-salient feature mining (NSFM) module is designed to extract mask and edge features from the image encoder to enrich the representation of the mask and edge prediction layers (i.e., MPL and EPL). During training, the parameters of the pre-trained SAM are frozen, with the newly added parameters in VNS-SAM trained. During inference, VNS-SAM outputs the more precise VNS-mask and the original SAM-mask. The prompt encoder and prompt tokens are omitted here.
• Figure 4: Details of Mask Prediction Layer (MPL) and Edge Prediction Layer (EPL). In MPL, we reuse the highly optimized image embeddings $\text{F}$ as supplementary features. (Best viewed in color)
• Figure 5: Details of Non-Salient Feature Mining (NSFM) module. The multi-level features extracted from the backbone are decomposed into different components. Then, the most informative high-frequency and low-frequency components are selected and multi-level features are aggregated for edge and mask feature extraction. (Best viewed in color)