Towards Visual Grounding: A Survey

TL;DR

This survey provides a comprehensive, structured review of visual grounding (REC/PG) from early CNN/LSTM methods to modern Transformer, VLP, and grounding-vision-language models, including generalized and open-world settings. It clarifies definitions, proposes standardized settings, and analyzes a wide range of datasets, benchmarks, and applications. Key contributions include a formal taxonomy of grounding settings, a synthesis of dataset dynamics and evaluation metrics, and a roadmap of challenges and future directions, such as self-supervised grounding pre-training and GMLLM-based grounding. The work aims to guide researchers and practitioners in designing fair comparisons, selecting appropriate benchmarks, and advancing toward generalized, scalable, and multi-modal grounding systems with real-world impact.

Abstract

Visual Grounding, also known as Referring Expression Comprehension and Phrase Grounding, aims to ground the specific region(s) within the image(s) based on the given expression text. This task simulates the common referential relationships between visual and linguistic modalities, enabling machines to develop human-like multimodal comprehension capabilities. Consequently, it has extensive applications in various domains. However, since 2021, visual grounding has witnessed significant advancements, with emerging new concepts such as grounded pre-training, grounding multimodal LLMs, generalized visual grounding, and giga-pixel grounding, which have brought numerous new challenges. In this survey, we first examine the developmental history of visual grounding and provide an overview of essential background knowledge. We systematically track and summarize the advancements, and then meticulously define and organize the various settings to standardize future research and ensure a fair comparison. Additionally, we delve into numerous related datasets and applications, and highlight several advanced topics. Finally, we outline the challenges confronting visual grounding and propose valuable directions for future research, which may serve as inspiration for subsequent researchers. By extracting common technical details, this survey encompasses the representative work in each subtopic over the past decade. To the best of our knowledge, this paper represents the most comprehensive overview currently available in the field of visual grounding. This survey is designed to be suitable for both beginners and experienced researchers, serving as an invaluable resource for understanding key concepts and tracking the latest research developments. We keep tracing related work at https://github.com/linhuixiao/Awesome-Visual-Grounding.

Figures (12)

• Figure 1: An illustration of visual grounding.
• Figure 2: The number of papers and performance trends of visual grounding over the past decade. The data in panel (a) are derived from an exact-match lookup on Google Scholar for the term "referring expression comprehension". The GMLLMs in (b) are the 7B version.
• Figure 3: A future-oriented definition of generalized grounding.
• Figure 4: Overview of the paper structure, detailing Chapter \ref{['sec:introduction']}-\ref{['sec:future_direction']}, and Appendix Chapter A2-A4.
• Figure 5: A chronological overview of the representative research progress in fully supervised visual grounding from the perspective of the technical roadmap (\ref{['subsubsec:road_map']}). The corresponding citations for abbreviated methods can be found in the main text.