Language-Guided Diffusion Model for Visual Grounding

TL;DR

LG-DVG reframes visual grounding as a generative, language-guided denoising process over bounding boxes, enabling iterative, query-aware refinement without relying on handcrafted region proposals. It combines a frozen visual backbone, a Phrase-Bert text encoder, and a cross-modal transformer-based grounding decoder to perform text-conditioned box regression within a diffusion framework, using a forward diffusion on boxes and a DDIM-based reverse process. The approach introduces phrase-balanced box proposals and a Hungarian matching-based training objective that couples box coordinates with cross-modal similarity, achieving competitive accuracy across five datasets and improved efficiency relative to several iterative baselines. The results demonstrate the practicality of diffusion models for robust cross-modal grounding and highlight the potential for generative, iterative reasoning in vision-language tasks.

Abstract

Visual grounding (VG) tasks involve explicit cross-modal alignment, as semantically corresponding image regions are to be located for the language phrases provided. Existing approaches complete such visual-text reasoning in a single-step manner. Their performance causes high demands on large-scale anchors and over-designed multi-modal fusion modules based on human priors, leading to complicated frameworks that may be difficult to train and overfit to specific scenarios. Even worse, such once-for-all reasoning mechanisms are incapable of refining boxes continuously to enhance query-region matching. In contrast, in this paper, we formulate an iterative reasoning process by denoising diffusion modeling. Specifically, we propose a language-guided diffusion framework for visual grounding, LG-DVG, which trains the model to progressively reason queried object boxes by denoising a set of noisy boxes with the language guide. To achieve this, LG-DVG gradually perturbs query-aligned ground truth boxes to noisy ones and reverses this process step by step, conditional on query semantics. Extensive experiments for our proposed framework on five widely used datasets validate the superior performance of solving visual grounding, a cross-modal alignment task, in a generative way. The source codes are available at https://github.com/iQua/vgbase/tree/main/examples/DiffusionVG.

Figures (16)

• Figure 1: Our proposed visual grounding framework with an iterative learning pipeline built upon the Markov Chain of diffusion models. The forward diffusion process $q$ diffuses query-aligned boxes from ground truth boxes $\mathbb{B}^0$ to ones $\mathbb{B}^T$ with random distribution, while $p$ is the reverse denoising process. Each reverse step relies on a trainable model $\theta$ to make predictions conditional on the query sentence $\bm{Q}$ - "Two men looking at a bulldozer." Two men" corresponds to two ground truth boxes.
• Figure 2: The language-guided noisy-to-box pipeline overview of our proposed LG-DVG. Given image $\bm{I}$ and the text query $\bm{Q}$ - "Three men and one woman are working on computers", LG-DVG begins by utilizing visual and text encoders to transform the input image and text queries into backbone features. Based on the vision backbone, features of noisy boxes$\mathbb{B}^t$ at time step $t$ generated from a set of proposal boxes can be directly extracted. Subsequently, the novel cross-modal transformer fuses box and text features to produce compact multi-modal representations. Eventually, conditioned on text representation, our language-guided grounding decoder optimized by a compound objective function performs box regression to progressively refine box predictions $\widehat{\mathbb{B}}^t$ from noisy boxes.
• Figure 3: Our proposed cross-modal Transformer. We omit the layer norm and feedforward layer AllAttention for brevity in this figure.
• Figure 4: Comparison of the progressive refinement property of LG-DVG on Flickr30k EntitiesFlickr30kE dataset using different visual and text encoders. For all cases, the accuracy increases with refinement times.
• Figure 5: Qualitative results illustrating how LG-DVG progressively refines noisy boxes ($T=5$) conditional on the language query to approach ground truth boxes ($T=1$). In the fifth column, we present the ground truth boxes as the black rectangle to compare with our predictions. The results presented in the third, fourth, and fifth rows derive from test sets of RefCOCO/google, RefCOCO+/unc, and refcocog/umd, respectively. For ease of presentation, we sample top $50$ boxes with the largest $\widehat{S}$ from $150$ proposal boxes.