WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation

TL;DR

WeakMCN targets weakly supervised grounding by jointly learning Referring Expression Comprehension and Segmentation in a dual-branch network. It introduces Dynamic Visual Feature Enhancement to adaptively fuse visual knowledge from multiple foundation models and Collaborative Consistency Module to align optimization across tasks via Spatial Consistency and Inconsistency Suppression losses. The model achieves state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg in both WREC and WRES, and demonstrates strong generalization under semi-supervised settings with minimal labels. These contributions reduce annotation burdens while improving cross-task grounding precision, with practical impact on real-world visual grounding tasks requiring weak supervision.

Abstract

Weakly supervised referring expression comprehension(WREC) and segmentation(WRES) aim to learn object grounding based on a given expression using weak supervision signals like image-text pairs. While these tasks have traditionally been modeled separately, we argue that they can benefit from joint learning in a multi-task framework. To this end, we propose WeakMCN, a novel multi-task collaborative network that effectively combines WREC and WRES with a dual-branch architecture. Specifically, the WREC branch is formulated as anchor-based contrastive learning, which also acts as a teacher to supervise the WRES branch. In WeakMCN, we propose two innovative designs to facilitate multi-task collaboration, namely Dynamic Visual Feature Enhancement(DVFE) and Collaborative Consistency Module(CCM). DVFE dynamically combines various pre-trained visual knowledge to meet different task requirements, while CCM promotes cross-task consistency from the perspective of optimization. Extensive experimental results on three popular REC and RES benchmarks, i.e., RefCOCO, RefCOCO+, and RefCOCOg, consistently demonstrate performance gains of WeakMCN over state-of-the-art single-task alternatives, e.g., up to 3.91% and 13.11% on RefCOCO for WREC and WRES tasks, respectively. Furthermore, experiments also validate the strong generalization ability of WeakMCN in both semi-supervised REC and RES settings against existing methods, e.g., +8.94% for semi-REC and +7.71% for semi-RES on 1% RefCOCO. The code is publicly available at https://github.com/MRUIL/WeakMCN.

Figures (7)

• Figure 1: Performance comparison between single-task baselines and our multi-task network (WeakMCN). WeakMCN not only unifies two tasks into a single network, but also obviously outperforms common single-task baselines.
• Figure 2: Comparison of previous methods and WeakMCN. In sub-figure (a) and (b), previous WeakREC and WeakRES often adopt the independent single-task modeling. In (c), WeakMCN is the first time to joint learn WeakREC and WeakRES in a collaborative way.
• Figure 3: The overall framework of WeakMCN. The referring expression is processed by a text encoder, while the image is processed by multiple foundation models and aggregated into a visual bank. The DVFE module dynamically retrieves features from this visual bank to support the WREC and WRES branches, which predict the target bounding box and segmentation mask, respectively. During training, contrastive loss and SAM-based pseudo-labeled mask loss are used, with the CCM module enhancing collaboration between both tasks.
• Figure 4: Overview of Dynamic Visual Feature Enhancement (DVFE). DVFE predicts two groups of weights to dynamically combine visual features for WREC and WRES.
• Figure 5: The Collaborative Consistency Module (CCM) architecture. It consists of a Spacial Consistency Loss (SCL) $L_{scl}$ and an Inconsistency Suppression Loss (ISL) $L_{inc}$.