Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

Minghang Zheng, Jiahua Zhang, Qingchao Chen, Yuxin Peng, Yang Liu

TL;DR

The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions.

Abstract

Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at https://github.com/minghangz/ResVG.

ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

TL;DR

The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions.

Abstract

Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at https://github.com/minghangz/ResVG.
Paper Structure (17 sections, 3 equations, 6 figures, 6 tables)

This paper contains 17 sections, 3 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Grounding
  4. Data Augmentation
  5. method
  6. TransVG Revisited
  7. Overview
  8. Relation-Sensitive Data Augmentation
  9. Semantic-Sensitive Visual Grounding
  10. Discussion
  11. Experiment
  12. Datasets
  13. Implementation Details
  14. Comparisons with State-of-the-art Methods
  15. Ablation Study
  16. ...and 2 more sections

Figures (6)

  • Figure 1: The performance and number of samples on the RefCOCO validation dataset when there are different numbers of objects of the same category with the target object in an image.
  • Figure 2: Our method comprises two key components: Relation-Sensitive Data Augmentation and Semantic-Sensitive Visual Grounding. Firstly, we augment training data with multiple instances of the same category, emphasizing spatial relationships through generated images and pseudo queries. Secondly, we inject fine-grained semantic information into the grounding model to enhance understanding of object semantics.
  • Figure 3: Comparison with the baseline TransVG deng2021transvg. Our method can better distinguish target objects of the same category but with different fine-grained attribute semantics.
  • Figure 4: The performance of different queries without and with spatial relationship description on RefCOCO validation dataset.
  • Figure 5: Four visualization examples are generated by the Relation-Sensitive Data Augmentation module.
  • ...and 1 more figures