Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supplementing Missing Visions via Dialog for Scene Graph Generations

Zhenghao Zhao, Ye Zhu, Xiaoguang Zhu, Yuzhang Shang, Yan Yan

TL;DR

This work tackles Scene Graph Generation under incomplete visual data by introducing Supplementary Interactive Dialog (SI-Dial), a model-agnostic framework that enables a vision system to ask and receive natural language questions to fill in missing information. The method formalizes SGG as a three-part process, with object detection on incomplete input, dialog-based refinement, and relation prediction, and augments the detector outputs using a cross-modal vision update guided by a 10-round QA dialog powered by Sentence-BERT question embeddings and a CNN-based question decoder. Extensive experiments on Visual Genome across three missingness levels show that SI-Dial can significantly improve SGG performance, particularly under semantic masking, while also revealing that some levels of missing vision cause only modest degradation, suggesting redundancy in visual information. The approach demonstrates a practical, privacy-aware direction for robust multimodal reasoning by leveraging interactive language to compensate for limited visual data, with broader implications for dialog-enabled AI systems in vision tasks.

Abstract

Most current AI systems rely on the premise that the input visual data are sufficient to achieve competitive performance in various computer vision tasks. However, the classic task setup rarely considers the challenging, yet common practical situations where the complete visual data may be inaccessible due to various reasons (e.g., restricted view range and occlusions). To this end, we investigate a computer vision task setting with incomplete visual input data. Specifically, we exploit the Scene Graph Generation (SGG) task with various levels of visual data missingness as input. While insufficient visual input intuitively leads to performance drop, we propose to supplement the missing visions via the natural language dialog interactions to better accomplish the task objective. We design a model-agnostic Supplementary Interactive Dialog (SI-Dial) framework that can be jointly learned with most existing models, endowing the current AI systems with the ability of question-answer interactions in natural language. We demonstrate the feasibility of such a task setting with missing visual input and the effectiveness of our proposed dialog module as the supplementary information source through extensive experiments and analysis, by achieving promising performance improvement over multiple baselines.

Supplementing Missing Visions via Dialog for Scene Graph Generations

TL;DR

This work tackles Scene Graph Generation under incomplete visual data by introducing Supplementary Interactive Dialog (SI-Dial), a model-agnostic framework that enables a vision system to ask and receive natural language questions to fill in missing information. The method formalizes SGG as a three-part process, with object detection on incomplete input, dialog-based refinement, and relation prediction, and augments the detector outputs using a cross-modal vision update guided by a 10-round QA dialog powered by Sentence-BERT question embeddings and a CNN-based question decoder. Extensive experiments on Visual Genome across three missingness levels show that SI-Dial can significantly improve SGG performance, particularly under semantic masking, while also revealing that some levels of missing vision cause only modest degradation, suggesting redundancy in visual information. The approach demonstrates a practical, privacy-aware direction for robust multimodal reasoning by leveraging interactive language to compensate for limited visual data, with broader implications for dialog-enabled AI systems in vision tasks.

Abstract

Most current AI systems rely on the premise that the input visual data are sufficient to achieve competitive performance in various computer vision tasks. However, the classic task setup rarely considers the challenging, yet common practical situations where the complete visual data may be inaccessible due to various reasons (e.g., restricted view range and occlusions). To this end, we investigate a computer vision task setting with incomplete visual input data. Specifically, we exploit the Scene Graph Generation (SGG) task with various levels of visual data missingness as input. While insufficient visual input intuitively leads to performance drop, we propose to supplement the missing visions via the natural language dialog interactions to better accomplish the task objective. We design a model-agnostic Supplementary Interactive Dialog (SI-Dial) framework that can be jointly learned with most existing models, endowing the current AI systems with the ability of question-answer interactions in natural language. We demonstrate the feasibility of such a task setting with missing visual input and the effectiveness of our proposed dialog module as the supplementary information source through extensive experiments and analysis, by achieving promising performance improvement over multiple baselines.
Paper Structure (9 sections, 5 equations, 1 figure, 1 table)

This paper contains 9 sections, 5 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Task Formulations
  4. SI-Dial for Missing Visions
  5. Pipeline Training
  6. Experiments
  7. Experimental Setup
  8. Scene Graph Generation Results
  9. Conclusion

Figures (1)

  • Figure 1: The overall architecture of our proposed SI-Dial framework. We first obtain the preliminary objects from the object detector based on the incomplete visual input, and propose to conduct an interactive dialog process. Note that the dashed lines denote the operations only after the dialog is completed) for the final scene graph generation.