Learning Visual Grounding from Generative Vision and Language Model

TL;DR

This work demonstrates that generative vision-language models pre-trained on image-text data can provide object-level grounding knowledge when prompted, enabling scalable visual grounding without human-annotated queries. By prompting PaLI-3 to generate region captions for detection-box crops and enriching them with spatial-relational and attribute information, the authors construct VLM-VG, a large-scale dataset with 512K images, 1.1M objects, and 16.2M referring expressions, all model-generated. A lightweight grounding model trained on VLM-VG achieves state-of-the-art zero-shot performance on REC and RES benchmarks (RefCOCO/+/g), outperforming methods trained on manually labeled grounding data. The results suggest that leveraging generative VLMs can substantially reduce labeling costs while maintaining high grounding quality, with practical implications for real-world visual grounding systems.

Abstract

Visual grounding tasks aim to localize image regions based on natural language references. In this work, we explore whether generative VLMs predominantly trained on image-text data could be leveraged to scale up the text annotation of visual grounding data. We find that grounding knowledge already exists in generative VLM and can be elicited by proper prompting. We thus prompt a VLM to generate object-level descriptions by feeding it object regions from existing object detection datasets. We further propose attribute modeling to explicitly capture the important object attributes, and spatial relation modeling to capture inter-object relationship, both of which are common linguistic pattern in referring expression. Our constructed dataset (500K images, 1M objects, 16M referring expressions) is one of the largest grounding datasets to date, and the first grounding dataset with purely model-generated queries and human-annotated objects. To verify the quality of this data, we conduct zero-shot transfer experiments to the popular RefCOCO benchmarks for both referring expression comprehension (REC) and segmentation (RES) tasks. On both tasks, our model significantly outperform the state-of-the-art approaches without using human annotated visual grounding data. Our results demonstrate the promise of generative VLM to scale up visual grounding in the real world. Code and models will be released.

Figures (6)

• Figure 1: Overview of referring expression generation. We propose to generate grounding annotations automatically using generative vision language models. We construct three types of annotations to model the diversity of human linguistic norms. Type 1: regional caption directly generated by prompting the VLM with a generic instruction to describe the major object in the cropped image regions. Type 2: relational descriptions generated by rule-based methods utilizing spatial heuristics. Type 3: Attribute-rich descriptions that explicitly model attributes of the object by querying the VLM with "attribute” prompts.
• Figure 2: Visualization of VLM-VG dataset. By human examination, the incorrect or inaccurate annotations are colored red.
• Figure 3: Diversity of generated annotations. Our VLM-VG dataset provides referring expressions annotations from multiple perspectives aligning with human linguistic manners.
• Figure 4: Visualization of the zero-sot REC and RES predictions on RefCOCO and RefCOCO+. RefCOCO dataset requires spatial relationship understanding.
• Figure 5: Visualization of the zero-sot REC and RES predictions on RefCOCOg. RefCOCOg requires models to understand longer and more complex referring expressions.