Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension

TL;DR

This work tackles weakly-supervised referring image segmentation by learning from image-text pairs. It introduces PCNet, which uses a large language model to decompose a referring description into multiple target-related cues and then progressively refines cross-modal alignment through a multi-stage Conditional Referring Module. Two novel losses, Region-aware Shrinking and Instance-aware Disambiguation, supervise progressive localization and disambiguation among overlapping instance maps. Across three standard benchmarks, PCNet achieves state-of-the-art results, highlighting the effectiveness of progressive textual comprehension for fine-grained visual grounding.

Abstract

This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object. Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner. Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages. Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

Figures (15)

• Figure 1: Given an image and a language description as inputs (a), RIS aims to predict the target object (d). Unlike existing methods (e.g., TRIS liu2023referring (e) -- a WRIS method) that directly utilize the complete language description for target localization, we observe that humans would naturally break down the sentence into several key cues (e.g., Q1 -- Q3) and progressively converge onto the target object (from (b) to (d). This behavior inspires us to develop the Progressive Comprehension Network (PCNet), which merges text cues pertinent to the target object step-by-step (from (f) to (h)), significantly enhancing visual localization. $\oplus$ denotes the text combination operation.
• Figure 2: The pipeline of PCNet. Given a pair of image-text as input, PCNet enhances the visual-linguistic alignment by progressively comprehending the target-related textual nuances in the text description. It starts with using a LLM to decompose the input description into several target-related short phrases as target-related textual cues. The proposed Conditional Referring Module (CRM) then processes these cues to update the linguistic embeddings across multiple stages. Two novel loss functions, Region-aware Shrinking (RaS) and Instance-aware Disambiguation (IaD), are also proposed to supervise the progressive comprehension process.
• Figure 3: qVisual results of our PCNet. The green markers denote the peaks of the response maps.
• Figure 4: Visualization of the ablation study to show the efficacy of each proposed component.
• Figure 5: A failure case of our PCNet. As our model design assumes that there is only one object referred to by the language expression, it usually returns only one object.