Adapting the Segment Anything Model During Usage in Novel Situations

TL;DR

This work investigates the Segment Anything Model (SAM) for interactive segmentation in domains far from its core training data. It introduces a test-time adaptation framework that updates only a lightweight decoder using pseudo-labels derived from user clicks and eroded final masks, preserving real-time interactivity. The approach yields substantial reductions in failure rates across multiple rare-object and medical-image datasets (e.g., up to a 48.1% relative reduction in $FR_{20}@85$ and 46.6% in $FR_{30}@90$), while keeping computational overhead minimal. This enables immediate applicability of SAM to specialized domains without requiring additional domain-specific data or full fine-tuning, broadening the practical impact of foundation models in interactive segmentation.

Abstract

The interactive segmentation task consists in the creation of object segmentation masks based on user interactions. The most common way to guide a model towards producing a correct segmentation consists in clicks on the object and background. The recently published Segment Anything Model (SAM) supports a generalized version of the interactive segmentation problem and has been trained on an object segmentation dataset which contains 1.1B masks. Though being trained extensively and with the explicit purpose of serving as a foundation model, we show significant limitations of SAM when being applied for interactive segmentation on novel domains or object types. On the used datasets, SAM displays a failure rate $\text{FR}_{30}@90$ of up to $72.6 \%$. Since we still want such foundation models to be immediately applicable, we present a framework that can adapt SAM during immediate usage. For this we will leverage the user interactions and masks, which are constructed during the interactive segmentation process. We use this information to generate pseudo-labels, which we use to compute a loss function and optimize a part of the SAM model. The presented method causes a relative reduction of up to $48.1 \%$ in the $\text{FR}_{20}@85$ and $46.6 \%$ in the $\text{FR}_{30}@90$ metrics.

Figures (2)

• Figure 1: A rough description of the SAM architecture and the information used as pseudo-labels. Our method only adapts the mask-decoder which renders the computational effort of the backpropagation and optimization negligible. The gradient computation is displayed in red. The usage of pseudo-labels is discussed in \ref{['sec:adaption']}.
• Figure 2: Examples for the masks occurring during the interaction. The first row contains the ground truth. The second row contains the annotated mask and the clicks. The third row contains examples for the eroded result mask. Green, red and blue correspond to foreground, background and the eroded area, respectively.