An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding

TL;DR

The paper tackles the quadratic cost of Transformer self-attention in cross-modal visual grounding by proposing EEVG, a Decoder-only MTVG framework that treats linguistic features as memory and visual features as queries to achieve linear scaling with language length. It combines a parameter-free token-elimination strategy to prune background visual tokens with a lightweight MLP-based mask head that directly maps sparse tokens to segmentation masks, enabling end-to-end multi-task training for REC and RES. The approach yields state-of-the-art results on RefCOCO, RefCOCO+, and especially RefCOCOg, while delivering substantial speedups (e.g., ~28.19% FPS over PolyFormer) and strong generalization on GRES. These innovations make visual grounding more scalable to long-context dialogs and open-vocabulary scenarios, with practical implications for real-time and resource-constrained systems.

Abstract

Most advanced visual grounding methods rely on Transformers for visual-linguistic feature fusion. However, these Transformer-based approaches encounter a significant drawback: the computational costs escalate quadratically due to the self-attention mechanism in the Transformer Encoder, particularly when dealing with high-resolution images or long context sentences. This quadratic increase in computational burden restricts the applicability of visual grounding to more intricate scenes, such as conversation-based reasoning segmentation, which involves lengthy language expressions. In this paper, we propose an efficient and effective multi-task visual grounding (EEVG) framework based on Transformer Decoder to address this issue, which reduces the cost in both language and visual aspects. In the language aspect, we employ the Transformer Decoder to fuse visual and linguistic features, where linguistic features are input as memory and visual features as queries. This allows fusion to scale linearly with language expression length. In the visual aspect, we introduce a parameter-free approach to reduce computation by eliminating background visual tokens based on attention scores. We then design a light mask head to directly predict segmentation masks from the remaining sparse feature maps. Extensive results and ablation studies on benchmarks demonstrate the efficiency and effectiveness of our approach. Code is available in https://github.com/chenwei746/EEVG.

Figures (10)

• Figure 1: Comparison of different frameworks: (a) Encoder-Decoder methods, (b) Encoder-only methods, and (c) our Decoder-only framework EEVG. In (a) and (b), Transformer Encoder is utilized, and all visual tokens are employed for mask generation. (c) Our method EEVG leverages the Transformer Decoder to integrate diverse modality information and remove background visual tokens during modalities fusion.
• Figure 2: Overview of our method. The language tokens and visual tokens are extracted by a linguistic backbone and a visual backbone which are not shown in the figure.
• Figure 3: Time complexity comparison between Encoder and Decoder. There is only one input for Encoder while Decoder has two inputs: query and memory. $N$ denotes the number of visual tokens, $L$ means the number of linguistic tokens, $C$ is the dimension of tokens, and "Add & Norm" refers to residual connection and normalization.
• Figure 4: We conduct the visual tokens elimination process in each Decoder layer. "ASA" denotes adaptive spatial attention and "Norm & Elimination" means normalization and eliminating visual tokens according to Eq. \ref{['eq:normalization']}.
• Figure 5: Visualization of attention scores $S_{loc}$ of location token and visual tokens. GT refers to ground truth, "w/o ASA" denotes without using adaptive spatial attention, and "$1^{st} \text{Layer Score}$" means the first Decoder layer attention scores between location token and visual tokens.