Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Query-guided Prototype Evolution Network for Few-Shot Segmentation

Runmin Cong, Hang Xiong, Jinpeng Chen, Wei Zhang, Qingming Huang, Yao Zhao

TL;DR

QPENet addresses the limitation of support-only prototype generation in Few-Shot Segmentation by introducing query-guided evolution for both foreground and background prototypes. The framework combines Pseudo-prototype Generation, Dual Prototype Evolution, Global Background Cleansing, and a Feature Filtering and Activation module to tailor prototypes to each query image and fuse them effectively with query features. Empirical results on Pascal-$5^i$ and COCO-$20^i$ demonstrate state-of-the-art performance, reinforced by comprehensive ablations that validate the contribution of each component. The approach offers a practical impact by enabling more accurate segmentation for unseen classes with minimal annotations, leveraging query information to refine representations.

Abstract

Previous Few-Shot Segmentation (FSS) approaches exclusively utilize support features for prototype generation, neglecting the specific requirements of the query. To address this, we present the Query-guided Prototype Evolution Network (QPENet), a new method that integrates query features into the generation process of foreground and background prototypes, thereby yielding customized prototypes attuned to specific queries. The evolution of the foreground prototype is accomplished through a \textit{support-query-support} iterative process involving two new modules: Pseudo-prototype Generation (PPG) and Dual Prototype Evolution (DPE). The PPG module employs support features to create an initial prototype for the preliminary segmentation of the query image, resulting in a pseudo-prototype reflecting the unique needs of the current query. Subsequently, the DPE module performs reverse segmentation on support images using this pseudo-prototype, leading to the generation of evolved prototypes, which can be considered as custom solutions. As for the background prototype, the evolution begins with a global background prototype that represents the generalized features of all training images. We also design a Global Background Cleansing (GBC) module to eliminate potential adverse components mirroring the characteristics of the current foreground class. Experimental results on the PASCAL-$5^i$ and COCO-$20^i$ datasets attest to the substantial enhancements achieved by QPENet over prevailing state-of-the-art techniques, underscoring the validity of our ideas.

Query-guided Prototype Evolution Network for Few-Shot Segmentation

TL;DR

QPENet addresses the limitation of support-only prototype generation in Few-Shot Segmentation by introducing query-guided evolution for both foreground and background prototypes. The framework combines Pseudo-prototype Generation, Dual Prototype Evolution, Global Background Cleansing, and a Feature Filtering and Activation module to tailor prototypes to each query image and fuse them effectively with query features. Empirical results on Pascal- and COCO- demonstrate state-of-the-art performance, reinforced by comprehensive ablations that validate the contribution of each component. The approach offers a practical impact by enabling more accurate segmentation for unseen classes with minimal annotations, leveraging query information to refine representations.

Abstract

Previous Few-Shot Segmentation (FSS) approaches exclusively utilize support features for prototype generation, neglecting the specific requirements of the query. To address this, we present the Query-guided Prototype Evolution Network (QPENet), a new method that integrates query features into the generation process of foreground and background prototypes, thereby yielding customized prototypes attuned to specific queries. The evolution of the foreground prototype is accomplished through a \textit{support-query-support} iterative process involving two new modules: Pseudo-prototype Generation (PPG) and Dual Prototype Evolution (DPE). The PPG module employs support features to create an initial prototype for the preliminary segmentation of the query image, resulting in a pseudo-prototype reflecting the unique needs of the current query. Subsequently, the DPE module performs reverse segmentation on support images using this pseudo-prototype, leading to the generation of evolved prototypes, which can be considered as custom solutions. As for the background prototype, the evolution begins with a global background prototype that represents the generalized features of all training images. We also design a Global Background Cleansing (GBC) module to eliminate potential adverse components mirroring the characteristics of the current foreground class. Experimental results on the PASCAL- and COCO- datasets attest to the substantial enhancements achieved by QPENet over prevailing state-of-the-art techniques, underscoring the validity of our ideas.
Paper Structure (21 sections, 19 equations, 6 figures, 5 tables)

This paper contains 21 sections, 19 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Semantic Segmentation
  4. Few-Shot Segmentation
  5. Proposed Method
  6. Problem Definition
  7. Overview
  8. Foreground Prototype Evolution
  9. Pseudo-prototype Generation Module
  10. Dual Prototype Evolution Module
  11. Background Prototype Evolution
  12. Feature Filtering and Activation Module
  13. Training Loss
  14. Experiments
  15. Dataset and Evaluation Metric
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Comparison of prototype generation strategies in existing prototype-based methods and our QPENet. (a) The strategy in existing prototype-based methods that only considers the support image. (b) The strategy in our QPENet that considers both the support and query images. (c) A qualitative comparison of the previous prototype-based method tian2020prior and our QPENet.
  • Figure 2: Overall architecture of our proposed QPENet. Initially, it extracts features from both the support and query images using a pre-trained backbone. Subsequently, a foreground prototype evolution process, composed of the PPG and DPE modules, facilitates the creation of a foreground prototype specifically tailored to the current query. Concurrently, a background prototype evolution process, executed by the GBC module, eliminates potential components reflecting the current foreground class from the global background prototype, yielding a customized background prototype. Finally, the FFA module takes the query feature and multiple foreground and background prototypes as input, generating the final prediction.
  • Figure 3: The detailed architecture of the GBC module. It initially uses the global background prototype to segment the backgrounds of both the support and query images. With the assistance of the predicted query background mask, it then extracts the background features of the query. Finally, these background features enable the generation of a customized background prototype tailored to the specific query after undergoing fully connected layers.
  • Figure 4: Qualitative comparison against state-of-the-art methods in various representative scenes. From top to bottom: annotated support image; annotated query image; predictions of PFENet; predictions of CyCTR; predictions of NERTNet; predictions of our QPENet.
  • Figure 5: Qualitative results for component analysis. (a) Annotated support image. (b) Annotated query image. (c) Predictions of the baseline model. (d) Predictions of the baseline model enhanced by FGPE. (e) Predictions of the baseline model enhanced by FGPE and BGPE. (f) Predictions of the full model.
  • ...and 1 more figures