Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference

Kai Shen, Lingfei Wu, Siliang Tang, Fangli Xu, Bo Long, Yueting Zhuang, Jian Pei

TL;DR

This paper tackles visual question generation by addressing two core issues: the one-to-many image-to-question mappings and the need to model rich object relations. It introduces double hints—textual answers and visual regions—and a self-learning auto-encoder to derive visual hints without extra annotations, then forms a dynamic object graph learned end-to-end. The DH-Graph2Seq framework casts VQG as a graph-to-sequence problem, learning graph topology from double-hint embeddings and generating questions via separate attention over visual hints and the graph, with LSTM or Transformer decoders. Experiments on VQA2.0 and COCO-QA show significant improvements over strong baselines and demonstrate benefits for data augmentation and zero-shot VQA, indicating practical impact for VQG-enabled AI systems.

Abstract

The visual question generation (VQG) task aims to generate human-like questions from an image and potentially other side information (e.g. answer type). Previous works on VQG fall in two aspects: i) They suffer from one image to many questions mapping problem, which leads to the failure of generating referential and meaningful questions from an image. ii) They fail to model complex implicit relations among the visual objects in an image and also overlook potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. Concretely, we aim to ask the right visual questions with Double Hints - textual answers and visual regions of interests, which could effectively mitigate the existing one-to-many mapping issue. Particularly, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework, which first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilizes a graph-to-sequence model to generate the questions with double hints. Experimental results demonstrate the priority of our proposed method.

Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference

TL;DR

This paper tackles visual question generation by addressing two core issues: the one-to-many image-to-question mappings and the need to model rich object relations. It introduces double hints—textual answers and visual regions—and a self-learning auto-encoder to derive visual hints without extra annotations, then forms a dynamic object graph learned end-to-end. The DH-Graph2Seq framework casts VQG as a graph-to-sequence problem, learning graph topology from double-hint embeddings and generating questions via separate attention over visual hints and the graph, with LSTM or Transformer decoders. Experiments on VQA2.0 and COCO-QA show significant improvements over strong baselines and demonstrate benefits for data augmentation and zero-shot VQA, indicating practical impact for VQG-enabled AI systems.

Abstract

The visual question generation (VQG) task aims to generate human-like questions from an image and potentially other side information (e.g. answer type). Previous works on VQG fall in two aspects: i) They suffer from one image to many questions mapping problem, which leads to the failure of generating referential and meaningful questions from an image. ii) They fail to model complex implicit relations among the visual objects in an image and also overlook potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. Concretely, we aim to ask the right visual questions with Double Hints - textual answers and visual regions of interests, which could effectively mitigate the existing one-to-many mapping issue. Particularly, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework, which first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilizes a graph-to-sequence model to generate the questions with double hints. Experimental results demonstrate the priority of our proposed method.
Paper Structure (38 sections, 22 equations, 9 figures, 4 tables)

This paper contains 38 sections, 22 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Question Generation
  4. Graph-to-Sequence Learning
  5. Method
  6. Problem Formulation
  7. Image and Answer Encoder
  8. Double-Hints Guided Multi-Task Auto-Encoder
  9. Cross-modal Alignments
  10. Multi-Task Decoder
  11. Visual hint prediction
  12. Object position prediction
  13. Target answer prediction
  14. Double-Hints Guided Graph Construction and Embedding Learning
  15. Double-Hints Guided Question Generation
  16. ...and 23 more sections

Figures (9)

  • Figure 1: An example of our new setting for learning VQG with Double Hints: textual answers and visual regions of interests.
  • Figure 2: An example of the correlations among the visual objects.
  • Figure 3: The overall framework of our proposed model with double hints to guide VQG.
  • Figure 4: Case study (left) and error analysis (right) results.
  • Figure 5: The details of case study examples. The red rectangles mean the visual-hint regions.
  • ...and 4 more figures