Learning from Synthetic Data for Visual Grounding

TL;DR

This work investigates whether synthetic data, generated via text-to-image models, large-language models, and open-vocabulary detectors, can meaningfully improve visual grounding—the task of linking text to image regions. The authors introduce SynGround, a pipeline that yields image-text pairs and image-text-box triplets, and they demonstrate systematic gains on RefCOCO+ and Flickr30k across ALBEF and BLIP baselines. Key findings show that image-text pairs generated via Image2Text with detailed prompts and LLM-derived phrases are most effective, while synthetic image-text-box data with short phrases provides robust improvements, and synthetic data can augment or even surpass web-crawled data in certain settings. The approach scales to large synthetic datasets and remains competitive when real data is limited, offering a practical route to scalable visual grounding in multi-modal systems.

Abstract

This paper extensively investigates the effectiveness of synthetic training data to improve the capabilities of vision-and-language models for grounding textual descriptions to image regions. We explore various strategies to best generate image-text pairs and image-text-box triplets using a series of pretrained models under different settings and varying degrees of reliance on real data. Through comparative analyses with synthetic, real, and web-crawled data, we identify factors that contribute to performance differences, and propose SynGround, an effective pipeline for generating useful synthetic data for visual grounding. Our findings show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models and offers the potential for arbitrarily large scale data generation. Particularly, data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively, across the RefCOCO+ and the Flickr30k benchmarks.

Figures (14)

• Figure 1: Various approaches for obtaining text descriptions to use as input prompts for an image generator, resulting in image-text pairs. 1) Concatenation: Descriptions are generated by concatenating real text, 2) Text2Text: LLM summary on real text, and 3) Image2Text: Descriptions are produced from an image captioner.
• Figure 2: On the left, an overview of our SynGround image-text-box synthesis pipeline, and on the right some sample generated image-text-box triplets. We use an image description generator $\Psi_c$ to output a description that serves as a prompt to an image generator $\Psi_g$ to obtain synthetic image $I$. This description is also used to obtain text phrases $T$ by prompting an LLM $\Psi_t$. Finally, the synthetic text and image are fed into an object detector $\Psi_d$ to obtain synthetic boxes $B$.
• Figure 3: Two approaches for generating image descriptions ($\Psi_C$) for image synthesis and phrase extraction. The top pipeline, Image2Text, relies more on real data, applying an image captioning model to real images. The bottom pipeline, Context2Text, samples concepts from a predefined list and uses an LLM with in-context learning to generate image descriptions.
• Figure 4: Pointing game accuracy improvement on RefCOCO+ and Flickr30k at various scales. The line denotes the mean improvement across 3 sampled subsets at each scale, and the error bars are corresponding standard deviations.
• Figure 5: Random examples from the in-context learning database. The query "Q" contains two nouns, while the expected answer "A" is a crafted image description incorporating the queried nouns.