Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement

TL;DR

The paper tackles the challenge of adapting few-shot segmentation (FSS) models to novel classes with limited annotated data. It introduces Adaptive FSS, a plug-in framework that inserts a Prototype Adaptive Module (PAM) into the encoder to produce class-specific, multi-stage features, using a momentum-updated Prototype Bank and a lightweight Learnable Adaptive Module to refine representations. PEM constructs and enhances class prototypes, while LAM performs task-specific refinement, and only PAM is fine-tuned, enabling efficient adaptation across diverse backbones and FSS methods. Empirical results on PASCAL-5i and COCO-20i across four strong FSS models show state-of-the-art performance gains in 1-shot and 5-shot settings, as well as robustness to domain shift and limited data. The work offers a general, parameter-efficient approach to novel class adaptation in FSS with practical implications for rapid deployment of segmentation models in new domains.

Abstract

The Few-Shot Segmentation (FSS) aims to accomplish the novel class segmentation task with a few annotated images. Current FSS research based on meta-learning focus on designing a complex interaction mechanism between the query and support feature. However, unlike humans who can rapidly learn new things from limited samples, the existing approach relies solely on fixed feature matching to tackle new tasks, lacking adaptability. In this paper, we propose a novel framework based on the adapter mechanism, namely Adaptive FSS, which can efficiently adapt the existing FSS model to the novel classes. In detail, we design the Prototype Adaptive Module (PAM), which utilizes accurate category information provided by the support set to derive class prototypes, enhancing class-specific information in the multi-stage representation. In addition, our approach is compatible with diverse FSS methods with different backbones by simply inserting PAM between the layers of the encoder. Experiments demonstrate that our method effectively improves the performance of the FSS models (e.g., MSANet, HDMNet, FPTrans, and DCAMA) and achieve new state-of-the-art (SOTA) results (i.e., 72.4\% and 79.1\% mIoU on PASCAL-5$^i$ 1-shot and 5-shot settings, 52.7\% and 60.0\% mIoU on COCO-20$^i$ 1-shot and 5-shot settings). Our code can be available at https://github.com/jingw193/AdaptiveFSS.

Figures (4)

• Figure 1: The overview of our Adaptive FSS. Previous works generally train the FSS model on the base classes and directly evaluate it on novel classes. In our framework, we insert the Prototype Adaptive Module (PAM), conducting a fine-tuning step before testing to effectively adapt the model to novel classes through prototype enhancement.
• Figure 2: The overall architecture of our proposed Adaptive FSS. Given a support set $\{I_s, M_s\}$, the image $I_s$ is fed into the encoder and generates feature $F_s$ ($F_q$). In each PAM, with calculation between $F_s$ and mask $M_s$, the temporary prototype $P_t$ is first obtained to select prototype $P_i$ and update the bank. After that, the corresponding class prototype $P_i$ and feature $F_s$ ($F_q$) are combined to generate the class-specific feature $F_s^*$ ($F_q^*$). Finally, $F_s^*$ ($F_q^*$) is sent into the Learnable Adaptive Module, leading to the acquired $\hat{F_s}$ ($\hat{F_q}$), which are injected into the encoder.
• Figure 3: The visual comparison between baseline and our proposed Adaptive FSS on PASCAL-5$^i$ in 1-shot setting.
• Figure 4: The visual comparison between baseline and our proposed Adaptive FSS on PASCAL-5$^i$ in 1-shot setting.