Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fully Aligned Network for Referring Image Segmentation

Yong Liu, Ruihao Xu, Yansong Tang

TL;DR

A Fully Aligned Network (FAN) that follows four cross-modal interaction principles is presented, which achieves state-of-the-art performance on the prevalent RIS benchmarks (RefCOCO, RefCOCO+, G-Ref) with a simple architecture.

Abstract

This paper focuses on the Referring Image Segmentation (RIS) task, which aims to segment objects from an image based on a given language description. The critical problem of RIS is achieving fine-grained alignment between different modalities to recognize and segment the target object. Recent advances using the attention mechanism for cross-modal interaction have achieved excellent progress. However, current methods tend to lack explicit principles of interaction design as guidelines, leading to inadequate cross-modal comprehension. Additionally, most previous works use a single-modal mask decoder for prediction, losing the advantage of full cross-modal alignment. To address these challenges, we present a Fully Aligned Network (FAN) that follows four cross-modal interaction principles. Under the guidance of reasonable rules, our FAN achieves state-of-the-art performance on the prevalent RIS benchmarks (RefCOCO, RefCOCO+, G-Ref) with a simple architecture.

Fully Aligned Network for Referring Image Segmentation

TL;DR

A Fully Aligned Network (FAN) that follows four cross-modal interaction principles is presented, which achieves state-of-the-art performance on the prevalent RIS benchmarks (RefCOCO, RefCOCO+, G-Ref) with a simple architecture.

Abstract

This paper focuses on the Referring Image Segmentation (RIS) task, which aims to segment objects from an image based on a given language description. The critical problem of RIS is achieving fine-grained alignment between different modalities to recognize and segment the target object. Recent advances using the attention mechanism for cross-modal interaction have achieved excellent progress. However, current methods tend to lack explicit principles of interaction design as guidelines, leading to inadequate cross-modal comprehension. Additionally, most previous works use a single-modal mask decoder for prediction, losing the advantage of full cross-modal alignment. To address these challenges, we present a Fully Aligned Network (FAN) that follows four cross-modal interaction principles. Under the guidance of reasonable rules, our FAN achieves state-of-the-art performance on the prevalent RIS benchmarks (RefCOCO, RefCOCO+, G-Ref) with a simple architecture.
Paper Structure (18 sections, 2 figures, 3 tables)

This paper contains 18 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Image and Language Encoding
  6. Activation Module
  7. Vision-to-Language Decoder
  8. Language-to-Vision Decoder
  9. Discussion of Framework and Principles
  10. Experiment
  11. Datasets and Metrics.
  12. Implementation Details
  13. Comparison with State-of-the-arts
  14. Ablation Study
  15. Interaction Principles.
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Pipeline of our FAN. Taking an image and the corresponding language expression as input, the vision and language encoder extract corresponding features, respectively. Then a multi-scale activation module performs preliminary fusion between them to highlight the referred region roughly. For the decoding process, we update visual and linguistic features simultaneously to project them into the common space. Finally, the output mask is obtained by simple similarity calculation and binarization.
  • Figure 2: The structure of the Vision Projection Module (VPM).