OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

TL;DR

This paper proposes OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces and introduces a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling.

Abstract

Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef.

Figures (9)

• Figure 1: Comparison between our proposed approach and the mainstream REC/RES architectures.
• Figure 2: Illustration of our multimodal Mask Referring Modeling (MRefM) paradigm, which includes Referring-aware mask image modeling and Referring-aware mask language modeling.
• Figure 3: Illustration of the referring-based grounding and segmentation transfer.
• Figure 4: Illustrations of random masking (MAE) he2022masked, block-wise masking (BEiT) bao2021beit, and our referring-aware dynamic masking. $\alpha$ denotes the entire masking ratio, while $\beta$ and $\gamma$ denote the masking ratio beyond and within the referred region.
• Figure 5: The reconstruction of the visual target-relation score $\bm{{\bm{s}}}^{vt} \in \mathbb{R}^{N_v \times 4}$. $(x,y)$ represents the coordinates of a general image patch, and $P$ is the patch size. By slicing the predicted score, four masks can be derived. The score represents the spatial distance and the relative size between the current patch region and the referred region.