Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Dialogue State Tracking for Playing GuessWhich Game

Wei Pang, Ruixue Duan, Jinfu Yang, Ning Li

TL;DR

This paper tackles GuessWhich where a Questioner Bot must identify a hidden image by asking visually grounded questions. It introduces a dialogue-state-tracking framework that maintains both textual and mental imagery representations, $<M^{(t)}_{e}, M^{(t)}_{v}>$, and uses a Recursive Self-Reference Equation (R-SRE) and Visual Reasoning on Dialogue State (VRDS) to perform cross-modal reasoning and generate questions. The QEnc uses ViLBERT as an encoder; STrack employs Gumbel-Softmax to Add/Update states, and training optimizes $\,\mathcal{L}_{CE}$, $\,\mathcal{L}_{MSE}$, and $\mathcal{L}_{PL}$; experiments are conducted on VisDial v0.5, v0.9, and v1.0 showing state-of-the-art performance across metrics. Overall, the approach demonstrates a shift toward mentally grounded reasoning in visual dialogue, reducing reliance on real-image cues and providing publicly available code and data for reproducibility.

Abstract

GuessWhich is an engaging visual dialogue game that involves interaction between a Questioner Bot (QBot) and an Answer Bot (ABot) in the context of image-guessing. In this game, QBot's objective is to locate a concealed image solely through a series of visually related questions posed to ABot. However, effectively modeling visually related reasoning in QBot's decision-making process poses a significant challenge. Current approaches either lack visual information or rely on a single real image sampled at each round as decoding context, both of which are inadequate for visual reasoning. To address this limitation, we propose a novel approach that focuses on visually related reasoning through the use of a mental model of the undisclosed image. Within this framework, QBot learns to represent mental imagery, enabling robust visual reasoning by tracking the dialogue state. The dialogue state comprises a collection of representations of mental imagery, as well as representations of the entities involved in the conversation. At each round, QBot engages in visually related reasoning using the dialogue state to construct an internal representation, generate relevant questions, and update both the dialogue state and internal representation upon receiving an answer. Our experimental results on the VisDial datasets (v0.5, 0.9, and 1.0) demonstrate the effectiveness of our proposed model, as it achieves new state-of-the-art performance across all metrics and datasets, surpassing previous state-of-the-art models. Codes and datasets from our experiments are freely available at \href{https://github.com/xubuvd/GuessWhich}.

Multi-Modal Dialogue State Tracking for Playing GuessWhich Game

TL;DR

This paper tackles GuessWhich where a Questioner Bot must identify a hidden image by asking visually grounded questions. It introduces a dialogue-state-tracking framework that maintains both textual and mental imagery representations, , and uses a Recursive Self-Reference Equation (R-SRE) and Visual Reasoning on Dialogue State (VRDS) to perform cross-modal reasoning and generate questions. The QEnc uses ViLBERT as an encoder; STrack employs Gumbel-Softmax to Add/Update states, and training optimizes , , and ; experiments are conducted on VisDial v0.5, v0.9, and v1.0 showing state-of-the-art performance across metrics. Overall, the approach demonstrates a shift toward mentally grounded reasoning in visual dialogue, reducing reliance on real-image cues and providing publicly available code and data for reproducibility.

Abstract

GuessWhich is an engaging visual dialogue game that involves interaction between a Questioner Bot (QBot) and an Answer Bot (ABot) in the context of image-guessing. In this game, QBot's objective is to locate a concealed image solely through a series of visually related questions posed to ABot. However, effectively modeling visually related reasoning in QBot's decision-making process poses a significant challenge. Current approaches either lack visual information or rely on a single real image sampled at each round as decoding context, both of which are inadequate for visual reasoning. To address this limitation, we propose a novel approach that focuses on visually related reasoning through the use of a mental model of the undisclosed image. Within this framework, QBot learns to represent mental imagery, enabling robust visual reasoning by tracking the dialogue state. The dialogue state comprises a collection of representations of mental imagery, as well as representations of the entities involved in the conversation. At each round, QBot engages in visually related reasoning using the dialogue state to construct an internal representation, generate relevant questions, and update both the dialogue state and internal representation upon receiving an answer. Our experimental results on the VisDial datasets (v0.5, 0.9, and 1.0) demonstrate the effectiveness of our proposed model, as it achieves new state-of-the-art performance across all metrics and datasets, surpassing previous state-of-the-art models. Codes and datasets from our experiments are freely available at \href{https://github.com/xubuvd/GuessWhich}.
Paper Structure (5 sections, 9 equations, 3 figures, 2 tables)

This paper contains 5 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model
  4. Experiment and Evaluation
  5. Conclusion

Figures (3)

  • Figure 1: Illustration of three types of QBot involving four components: question (Q), answer (A), history encoder (H) at round t, and Decoder for new question generation. Specially, a) Non-visual QBot, no visual information is provided to QBot. b) Real-image QBot, retrieves a real image per round from a pool to serve as visual information for Decoder. c) Mental-imagery QBot (Ours), explores visually related reasoning based on the QBot's mental model of the secret image.
  • Figure 2: Overall structure of the proposed DST model. The oblong colorful strips denote image state $M^{(t)}_{v}$, and the rounded circle are word state $M^{(t)}_{e}$.
  • Figure 3: Comparison of generated dialogs with ADQ ADQ and humans on VisDial v1.0 val. Our QBot agent converses with SL-ABOT ADQ for a fair comparison to ADQ model.