Overcoming Support Dilution for Robust Few-shot Semantic Segmentation

TL;DR

This work tackles the dilution problem in Few-shot Semantic Segmentation, where increasing the number of supports dilutes high-contribution examples and degrades segmentation. The authors propose a three-part solution: a Contribution Index to quantify per-support usefulness, Symmetric Correlation to preserve high-contributed features under noise, and Support Image Pruning to form a compact, informative subset. Together, these components yield improved segmentation across COCO-20i and Pascal-5i, with strong cross-domain generalization and practical online/real-world demonstrations. The approach is modular and plug-in friendly, enabling robust improvements for existing FSS methods with scalable computation and real-world applicability.

Abstract

Few-shot Semantic Segmentation (FSS) is a challenging task that utilizes limited support images to segment associated unseen objects in query images. However, recent FSS methods are observed to perform worse, when enlarging the number of shots. As the support set enlarges, existing FSS networks struggle to concentrate on the high-contributed supports and could easily be overwhelmed by the low-contributed supports that could severely impair the mask predictions. In this work, we study this challenging issue, called support dilution, our goal is to recognize, select, preserve, and enhance those high-contributed supports in the raw support pool. Technically, our method contains three novel parts. First, we propose a contribution index, to quantitatively estimate if a high-contributed support dilutes. Second, we develop the Symmetric Correlation (SC) module to preserve and enhance the high-contributed support features, minimizing the distraction by the low-contributed features. Third, we design the Support Image Pruning operation, to retrieve a compact and high quality subset by discarding low-contributed supports. We conduct extensive experiments on two FSS benchmarks, COCO-20i and PASCAL-5i, the segmentation results demonstrate the compelling performance of our solution over state-of-the-art FSS approaches. Besides, we apply our solution for online segmentation and real-world segmentation, convincing segmentation results showing the practical ability of our work for real-world demonstrations.

Figures (15)

• Figure 1: When the number of supports gets larger, SOTA FSS method DCAMA shi2022dense (ECCV'22) cannot concentrate on the high-contributed supports and are distracted by the low-contributed supports. In this figure, we increase $N$ from 2 to 30, in the 30-support set, we omit some supports for briefness.
• Figure 2: Left: the performance of DCAMA cannot gain consistent improvements when the number of shots $N$ gets larger. Right: when mixing more noisy information, DCAMA ResNet-50 cannot protect the upper-bound support from dilution and we observe a drastic performance drop.
• Figure 3: The supports in the same category can have significant visual differences, different contributions to the query lead to different mask results.
• Figure 4: Left: The deviation value $\Delta$ v.s. the number of shots $N$. Right: segmentation mIoU v.s. the number of shots $N$. Experiments on COCO-20$^i$ Fold 1 under the upper-bound setting.
• Figure 5: The pipeline of our network. We introduce the flow ➀-➄ in Sec. \ref{['method pipeline']}. The most critical parts, ➁ and ➂, are developed against support dilution. ➁: Support Image Pruning (Sec. \ref{['method SIP']}) is deployed to select high-contributed supports (indexed by the $\circ$ icon) and abandon low-contributed supports (indexed by the $\times$ icon). ➂: A novel correlation module, Symmetric Correlation (Sec. \ref{['method SC']}), is multi-layer applied to preserve and enhance the high-contributed features.