TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation

TL;DR

The paper addresses cross-domain few-shot semantic segmentation by leveraging the Segment Anything Model (SAM) with a novel Task-Adaptive Visual Prompt (TAVP). It introduces multi-level feature fusion (MFF) to preserve low-level details and a Class Domain Task Adaptive Auto Prompt (CDTAP) module to disentangle class and domain information through prototype-based learning and learnable prompts, all within a lightweight fine-tuning framework. Experimental results across four CD-FSS benchmarks show consistent gains, including average improvements of $1.3\%$ in 1-shot and $11.76\%$ in 5-shot settings, demonstrating both effectiveness and efficiency relative to prior methods. The work highlights the potential of foundation-model transfer for CD-FSS and offers a practical pathway for automatic prompt generation and robust cross-domain adaptation.

Abstract

While large visual models (LVM) demonstrated significant potential in image understanding, due to the application of large-scale pre-training, the Segment Anything Model (SAM) has also achieved great success in the field of image segmentation, supporting flexible interactive cues and strong learning capabilities. However, SAM's performance often falls short in cross-domain and few-shot applications. Previous work has performed poorly in transferring prior knowledge from base models to new applications. To tackle this issue, we propose a task-adaptive auto-visual prompt framework, a new paradigm for Cross-dominan Few-shot segmentation (CD-FSS). First, a Multi-level Feature Fusion (MFF) was used for integrated feature extraction as prior knowledge. Besides, we incorporate a Class Domain Task-Adaptive Auto-Prompt (CDTAP) module to enable class-domain agnostic feature extraction and generate high-quality, learnable visual prompts. This significant advancement uses a unique generative approach to prompts alongside a comprehensive model structure and specialized prototype computation. While ensuring that the prior knowledge of SAM is not discarded, the new branch disentangles category and domain information through prototypes, guiding it in adapting the CD-FSS. Comprehensive experiments across four cross-domain datasets demonstrate that our model outperforms the state-of-the-art CD-FSS approach, achieving an average accuracy improvement of 1.3\% in the 1-shot setting and 11.76\% in the 5-shot setting.

Figures (8)

• Figure 1: (a) The existing cross-domain segmentation method based on SAM. (b) Our Task-adaptive Visual Prompt. (i) There is no interaction between features from different categories in the original feature distribution. (ii) In a specific domain, prototypes are used to make semantic distinctions between categories to achieve clustering. (iii) Finally, inter-category distinction and intra-category strong constraints are achieved in a unified space.
• Figure 2: The overall architecture of proposed TAVP network. First, the images from source domain with cut-mix are passed through the SAM encoder to obtain multi-level features, which are combined with original pre-trained weights on SA-1B dataset kirillov2023segment, and followed by a batch normalization layer to get the class-agnostic features. Additionally, CDTAP is employed for fine-tuning and meta-transformation. Simultaneously, dense embedding: $F_{prompt}$ and image embeddings are acquired as the input of the decoder. At last, the mask decoder predicts the query image. The $L_{dem}$ loss is used for learnable prompt supervision and fine-tuning, and $L_{seg}$ is used for supervising auto-prompts generation.
• Figure 3: Existing class-wise few-shot methods and our two-way matching meta-learning module.
• Figure 4: Details of MT and PG.
• Figure 5: Qualitative results of TAVP in 1-way 5-shot segmentation on CD-FSS. Support labels are overlaid in red. The ground truth and predictions of query images are highlighted, respectively.