BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Grounding

TL;DR

This work tackles bias and limited semantic reasoning in one-tower visual grounding. It introduces BARE, a bias-aware and reasoning-enhanced framework that preserves modality-specific cues and enables structured cross-modal interaction through three modules—Language Salience Modulator (LSM), Visual Bias Correction (VBC), and Referential Relationship Enhancement (R^2E)—along with Low-Rank Adaptation (LoRA) for efficient tuning. Empirically, BARE achieves state-of-the-art performance on five VG benchmarks and maintains favorable efficiency, underpinned by ablations that validate the complementary roles of LSM, VBC, and R^2E. The approach offers practical impact by improving grounding accuracy in linguistically complex and visually biased scenarios, with potential extensions to broader cross-modal tasks such as GREC and RIE.

Abstract

Visual Grounding (VG), which aims to locate a specific region referred to by expressions, is a fundamental yet challenging task in the multimodal understanding fields. While recent grounding transfer works have advanced the field through one-tower architectures, they still suffer from two primary limitations: (1) over-entangled multimodal representations that exacerbate deceptive modality biases, and (2) insufficient semantic reasoning that hinders the comprehension of referential cues. In this paper, we propose BARE, a bias-aware and reasoning-enhanced framework for one-tower visual grounding. BARE introduces a mechanism that preserves modality-specific features and constructs referential semantics through three novel modules: (i) language salience modulator, (ii) visual bias correction and (iii) referential relationship enhancement, which jointly mitigate multimodal distractions and enhance referential comprehension. Extensive experimental results on five benchmarks demonstrate that BARE not only achieves state-of-the-art performance but also delivers superior computational efficiency compared to existing approaches. The code is publicly accessible at https://github.com/Marloweeee/BARE.

Figures (11)

• Figure 1: Comparison of visual grounding frameworks. (a) Dual-tower architectures employ separate encoders for each modality but often suffer from grounding errors due to the modality gap. (b) One-tower architectures employ a shared encoder, which may lead to feature entanglement and limit grounding performance. (c) Our BARE leverages decoupled and fully interactive representations for accurate grounding.
• Figure 2: Visualization of layer-wise attention maps and grounding results from BEiT-3 and our proposed method. BARE effectively filters out deceptive visual shortcuts and progressively refines its focus toward the intended referent, leading to more precise and robust grounding under challenging conditions.
• Figure 3: Schematic overview of the BARE framework. The model extracts visual and textual features via dedicated tokenizers, with linguistic signals refined by the Language Salience Modulator (LSM). These features are then integrated within a modality-shared encoder that incorporates Visual Bias Correction (VBC) and Referential Relationship Enhancement (R$^2$E) modules to suppress modality-specific biases and reinforce compositional reasoning. Finally, a fusion encoder leverages a regression token ([REG]) to aggregate multimodal information for precise coordinate prediction.
• Figure 4: Architecture of the Language Salience Modulator (LSM). The module leverages a dual-gating mechanism to refine language signals: a salience gate accentuates task-critical cues, while a debiasing gate suppresses deceptive linguistic shortcuts via a learnable shared prior $l_{\text{bias}}$. The final representation is obtained by interpolating the original and debiased features.
• Figure 5: Detailed architectural variants of the proposed VBC modules. (a) The variant employing an alternative MHA between the semantic prototypes and features. (b) The variant that replaces MHA with a dot-product interaction parameterized by learnable weights. (c) The variant configured without semantic prototypes, utilizing an interaction scheme for feature processing.