HARIS: Human-Like Attention for Reference Image Segmentation

TL;DR

HARIS addresses the RIS challenge by proposing a Human-Like Attention mechanism that uses a feedback signal to refine multi-modal fusion and reduce irrelevant vision-language pairs. Coupled with a parameter-efficient fine-tuning (PEFT) framework, HARIS preserves zero-shot capabilities of pre-trained encoders while achieving state-of-the-art segmentation on RefCOCO, RefCOCO+, and G-Ref, and strong zero-shot performance on PhraseCut. The model employs a hierarchical design to fuse multi-scale visual features, and a Transformer-based mask decoder to generate precise masks, trained with a combination of focal and dice losses. Overall, HARIS advances RIS by improving object-centered attention and maintaining broad generalization with efficient adaptation.

Abstract

Referring image segmentation (RIS) aims to locate the particular region corresponding to the language expression. Existing methods incorporate features from different modalities in a \emph{bottom-up} manner. This design may get some unnecessary image-text pairs, which leads to an inaccurate segmentation mask. In this paper, we propose a referring image segmentation method called HARIS, which introduces the Human-Like Attention mechanism and uses the parameter-efficient fine-tuning (PEFT) framework. To be specific, the Human-Like Attention gets a \emph{feedback} signal from multi-modal features, which makes the network center on the specific objects and discard the irrelevant image-text pairs. Besides, we introduce the PEFT framework to preserve the zero-shot ability of pre-trained encoders. Extensive experiments on three widely used RIS benchmarks and the PhraseCut dataset demonstrate that our method achieves state-of-the-art performance and great zero-shot ability.

Figures (3)

• Figure 1: The attention maps and frameworks of existing methods and our method. Existing methods fuse features from different modalities based on the cross-attention mechanism, which follows a bottom-up manner, as shown in the left part of (c). As a result, the attention map generated by these methods may contain some irrelevant image-text pairs (the left part of (b)). For example, the correct region for 'black' is the left part of the image. However, existing methods also take the right part as relevant regions of 'black'. Different from existing methods, our method (the right part of (c)) introduces a feedback signal, which comes from modulated multi-modal tokens. Therefore, our method gets an accurate region for the word 'black' (the right part of (b)).
• Figure 2: The overview of HARIS. The input image is fed into the image encoder and outputs visual features ($F_{v_1},F_{v_2},F_{v_3}$) from different layers. Correspondingly, we send the language expression to the text encoder and get linguistic features $F_w$ and sentence representation $F_s$. Then, these features are sent into the Human-Like Attention blocks to get multi-modal features. Besides, we use a hierarchical architecture to use both semantic and grained visual features. Last, we send the multi-modal features and a learnable query token to the Transformer Decoder and get the final segmentation mask.
• Figure 3: The architecture of Human-Like Attention block. This block consists of two branches: Language-Weighted Attention and Vision-Weighted Attention. In the Language-Weighted Attention, we introduce the feedback signal, which is modulated by $F_s$ and fed into the MLP layer. Then, the feedback signal together with visual tokens acts as the second-round inputs for Language-Weighted Attention.