Collaborative Position Reasoning Network for Referring Image Segmentation

TL;DR

This work tackles Referring Image Segmentation by addressing the core challenge of explicit position reasoning. It introduces the Collaborative Positioning Reasoning Network (CPRN), which runs Row-and-Column interactive (RoCo) and Guided Holistic interactive (Holi) branches in parallel, with a global RoCo-to-Holi guidance pathway and a Feed Forward Network to fuse their outputs, followed by a Multi-Scale Decoder. RoCo decomposes visual features into directional row- and column-wise cues and interacts with language to produce a location prior, while Holi preserves holistic context with guided attention; together they yield precise referent localization and fine-grained segmentation. Extensive experiments on RefCOCO, RefCOCO+, and Gref demonstrate consistent state-of-the-art performance, with notable gains on small-scale objects and complex language expressions, underscoring the method’s effectiveness in explicit position reasoning for RIS.

Abstract

Given an image and a natural language expression as input, the goal of referring image segmentation is to segment the foreground masks of the entities referred by the expression. Existing methods mainly focus on interactive learning between vision and language to enhance the multi-modal representations for global context reasoning. However, predicting directly in pixel-level space can lead to collapsed positioning and poor segmentation results. Its main challenge lies in how to explicitly model entity localization, especially for non-salient entities. In this paper, we tackle this problem by executing a Collaborative Position Reasoning Network (CPRN) via the proposed novel Row-and-Column interactive (RoCo) and Guided Holistic interactive (Holi) modules. Specifically, RoCo aggregates the visual features into the row- and column-wise features corresponding two directional axes respectively. It offers a fine-grained matching behavior that perceives the associations between the linguistic features and two decoupled visual features to perform position reasoning over a hierarchical space. Holi integrates features of the two modalities by a cross-modal attention mechanism, which suppresses the irrelevant redundancy under the guide of positioning information from RoCo. Thus, with the incorporation of RoCo and Holi modules, CPRN captures the visual details of position reasoning so that the model can achieve more accurate segmentation. To our knowledge, this is the first work that explicitly focuses on position reasoning modeling. We also validate the proposed method on three evaluation datasets. It consistently outperforms existing state-of-the-art methods.

Figures (9)

• Figure 1: Comparison of visualization results between state-of-the-art methods and our proposed approach. The first row shows that the existing methods are prone to positioning errors on some Non-Salient targets with the small-scale. The second row shows that for some complex referring expressions, the reasoning ability of the previous models is not enough to position the target accurately.
• Figure 2: Conceptual Comparisons of typical architectures. (a) Simply combining the two modals of information. (b) Dividing the referring expression and fusing it with visual features progressively. (c) Exploring the relationship between visual and textual features by matching their attributes. (d) Adopting inference module in a multi-semantic-level progressively-fusion network architecture. $I$, $E$, $V$ and $L$ denotes image, natural language expression, visual feature and texture feature, respectively.
• Figure 3: Illustration of the difference between (a) previous methods and (b) our model. Previous works apply the holistic feature map in Cross-modal Attention. Differently, we use two parallel branches. The above one divides the holistic feature map into horizontal and vertical maps, and the lower one keeps the holistic feature map. They are fused with the textual feature, respectively, getting a better location for the referent.
• Figure 4: The overall architecture of our method. The CPRN block directly fuses features from two modal inputs of image $\bf{V}_{i}$ and text $\bf{L}$ to generate cross-modal feature representations. It includes RoCo interactive module and Holi interactive module. The multi-scale decoder generates segmentation results based on the interactive representation of multi-modal features at different stages.
• Figure 5: Illustration of our Collaborative Positioning Reasoning Network (CPRN). First, the visual features $\bf{V}_i$ generate vertical features and horizontal features via the Row-Column Aggregation. These two visual features are obtained through cross attention layers and it generates the multi-modal features with semantic information $\bf{v}_{h}^{hw}$ and $\bf{v}_{w}^{hw}$. Then, the final output of RoCo module are obtained through expand and addition operations. At the same time, $\bf{mask}_{roco}$ with positioning information is calculated, which guides the Holi module through the global guidance pathway for fine-grained segmentation. The Holi module utilizes $\bf{Mask}_{roco}$ and $\bf{Mask}_{holi}$ to generate the final output via a designed Guided Attention layer. Expand represents bilinear interpolation operation, and for convenience of representation, we ignore some residual connections.