VIPA: Visual Informative Part Attention for Referring Image Segmentation

TL;DR

The paper tackles Referring Image Segmentation by addressing cross-modal misalignment between vision queries and target information. It introduces Visual Informative Part Attention (VIPA), which treats informative visual parts as a Visual Expression serving as the key–value set in a Transformer-based segmentation decoder, rather than projecting visual context into language tokens. A Visual Expression Generator (VEG) retrieves and refines visual tokens using local-global linguistic cues and a masked cross-attention refinement, guided by a pixel-contrast loss, to produce semantically rich visual expressions. Across four RIS benchmarks, VIPA achieves state-of-the-art performance with encoder-agnostic robustness and improved efficiency compared with LLM-based RIS methods, demonstrating the practical impact of leveraging structured visual context for fine-grained segmentation.

Abstract

Referring Image Segmentation (RIS) aims to segment a target object described by a natural language expression. Existing methods have evolved by leveraging the vision information into the language tokens. To more effectively exploit visual contexts for fine-grained segmentation, we propose a novel Visual Informative Part Attention (VIPA) framework for referring image segmentation. VIPA leverages the informative parts of visual contexts, called a visual expression, which can effectively provide the structural and semantic visual target information to the network. This design reduces high-variance cross-modal projection and enhances semantic consistency in an attention mechanism of the referring image segmentation. We also design a visual expression generator (VEG) module, which retrieves informative visual tokens via local-global linguistic context cues and refines the retrieved tokens for reducing noise information and sharing informative visual attributes. This module allows the visual expression to consider comprehensive contexts and capture semantic visual contexts of informative regions. In this way, our framework enables the network's attention to robustly align with the fine-grained regions of interest. Extensive experiments and visual analysis demonstrate the effectiveness of our approach. Our VIPA outperforms the existing state-of-the-art methods on four public RIS benchmarks.

Figures (12)

• Figure 1: (a) Example of a transformer decoder used to extract language outputs. (b) Illustration of different RIS frameworks. Different from previous works, our approach leverages visual expression, generated from the retrieved informative parts of visual contexts, as a key-value set in the Transformer-based segmentation decoder. (c) Visual comparison of two different key-value sets. Yellow dotted boxes are incorrect predictions. The visual expression (VE) robustly guides the network's attention to the regions of interest by enhancing semantic coherence in the attention mechanism of referring image segmentation, whereas the advanced language expression (LE) results in incomplete predictions.
• Figure 2: Overview of Visual Informative Part Attention (VIPA) framework. VIPA robustly guides the network's attention to the region of interest by exploiting the visual expression generated from the retrieved informative parts of visual contexts.
• Figure 3: Qualitative comparison with the LLM-based RIS model lai2024lisa on RefCOCO+.
• Figure 4: Performance by increasing the value of $r$
• Figure 5: Segmentation results at different $r$