Learning Robust Correlation with Foundation Model for Weakly-Supervised Few-Shot Segmentation

TL;DR

The paper tackles weakly-supervised few-shot segmentation by eliminating the need for pixel-wise masks during training. It introduces CORENet, a foundation-model–assisted framework that fuses a correlation-guided transformer (CGT), a class-guided module (CGM), and an embedding-guided module (EGM) to learn robust support-query relations under mask noise, aided by Pixel-Adaptive Refinement and CLIP-driven category guidance. Core contributions include a multi-view correlation learning strategy, a coarse-to-fine category attention mechanism, and an embedding-based refinement that preserves appearance information. Empirical results on PASCAL-5i and COCO-20i demonstrate state-of-the-art performance in WS-FSS settings, with robust handling of unseen categories and noisy pseudo-masks, indicating strong practical potential for weak supervision in segmentation tasks.

Abstract

Existing few-shot segmentation (FSS) only considers learning support-query correlation and segmenting unseen categories under the precise pixel masks. However, the cost of a large number of pixel masks during training is expensive. This paper considers a more challenging scenario, weakly-supervised few-shot segmentation (WS-FSS), which only provides category ($i.e.$ image-level) labels. It requires the model to learn robust support-query information when the generated mask is inaccurate. In this work, we design a Correlation Enhancement Network (CORENet) with foundation model, which utilizes multi-information guidance to learn robust correlation. Specifically, correlation-guided transformer (CGT) utilizes self-supervised ViT tokens to learn robust correlation from both local and global perspectives. From the perspective of semantic categories, the class-guided module (CGM) guides the model to locate valuable correlations through the pre-trained CLIP. Finally, the embedding-guided module (EGM) implicitly guides the model to supplement the inevitable information loss during the correlation learning by the original appearance embedding and finally generates the query mask. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ have shown that CORENet exhibits excellent performance compared to existing methods.

Figures (6)

• Figure 1: Comparison between (a) few-shot segmentation (FSS) task shaban2017one, (b) weakly-supervised few-shot classification and segmentation (WS-FCS) task kang2023distilling, and (c) our weakly-supervised few-shot segmentation (WS-FSS) task settings. (a) The FSS task requires many support-query masks during training. (b) The classification and segmentation tasks are decoupled in the WS-FCS task. It provides supervisory information on whether images belong to the same category without providing specific category assistance for segmentation. (c) The WS-FSS task assists the model in segmentation through specific categories of supervised information in the presence of noise in the mask generated by the model.
• Figure 2: The overall architecture of our Correlation Enhancement Network (CORENet). Firstly, the Correlation-Guided Transformer (CGT) is introduced to generate robust correlation features using the local and global similarity calculations of ViT tokens. Then, with the assistance of CLIP, the Class-Guided Module (CGM) transforms the category information into coarse attention and further refines them to filter irrelevant information in the relevant features. Meanwhile, the Embedding-Guided Module (EGM) combines the support query appearance of each layer with the enhanced correlation features, further reducing the potential information loss of the model in correlation-enhanced learning under weakly-supervised settings and obtaining the final query mask.
• Figure 3: Illustration of Multi-kernel information fusion in CGT.
• Figure 4: Qualitative results of our CORENet on PASCAL-5$^i$ and COCO-20$^i$ benchmarks. Zoom in for details.
• Figure 5: Visualization of GradCAM obtained from different backbones of CLIP in CGM.