Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation

TL;DR

CRFormer tackles the challenge of preserving linguistic information during cross-modal propagation in referring image segmentation by introducing a Calibration Decoder that iteratively updates language features and a Language Reconstruction loss that supervises language fidelity. The method generates multiple image-specific language queries ($N_q$), fuses language into the vision backbone through Vision-Language Fusion, and uses a reconstruction objective to prevent language distortion. Empirical results on RefCOCO, RefCOCO+, and G-Ref show state-of-the-art performance across backbones, with ablations confirming the benefits of multi-query language representations, the calibration mechanism, and the reconstruction supervision. This approach advances robust cross-modal integration in RIS and offers a principled way to mitigate language degradation in deep transformer decoders.

Abstract

Referring image segmentation aims to segment an object referred to by natural language expression from an image. The primary challenge lies in the efficient propagation of fine-grained semantic information from textual features to visual features. Many recent works utilize a Transformer to address this challenge. However, conventional transformer decoders can distort linguistic information with deeper layers, leading to suboptimal results. In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder. We start by generating language queries using vision features, emphasizing different aspects of the input language. Then, we propose a novel Calibration Decoder (CDec) wherein the multi-modal features can iteratively calibrated by the input language features. In the Calibration Decoder, we use the output of each decoder layer and the original language features to generate new queries for continuous calibration, which gradually updates the language features. Based on CDec, we introduce a Language Reconstruction Module and a reconstruction loss. This module leverages queries from the final layer of the decoder to reconstruct the input language and compute the reconstruction loss. This can further prevent the language information from being lost or distorted. Our experiments consistently show the superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref datasets compared to state-of-the-art methods.

Figures (8)

• Figure 1: Distortion of language information propagation in the transformer decoder. (a) In the training process, the distortion of language information can still obtain the correct result in the image which has only two prominent targets by coincidence, but the traditional cross-entropy loss can not reflect this coincidence. (b) During the inference stage, the distortion of language information may result in inaccurate segmentation masks, especially in images featuring multiple targets, such as two girls with suitcases.
• Figure 2: The overview of our proposed CRFormer. Our method mainly consists of a text encoder, an image encoder, a Query Generation Module, a Calibration Decoder, and a Language Reconstruction Module.
• Figure 3: The details of Vision-Language Fusion Module.
• Figure 4: The details of Query Generation Module.
• Figure 5: The architecture of one block of our proposed Calibration Decoder (CDec).