TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

TL;DR

The paper extends visual question answering to the video domain by introducing TGIF-QA, a large-scale dataset with three new spatio-temporal tasks and a frame-QA task. It proposes ST-VQA, a dual-LSTM model with both spatial and temporal attention to capture cross-modal cues in videos, using ResNet-152 and C3D features and three decoding schemes. The authors show that video-based models and temporal attention outperform image-based baselines, and provide detailed dataset collection, quality control, and ablation analyses. This work advances video understanding in VQA and offers a valuable resource for spatio-temporal reasoning research, with code and data available on the project page.

Abstract

Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

Figures (5)

• Figure 1: Much of conventional VQA tasks focus on reasoning from images (top). This work proposes a new dataset with tasks designed specifically for video VQA that requires spatio-temporal reasoning from videos to answer questions correctly (bottom).
• Figure 2: Our TGIF-QA dataset introduces three new tasks for video QA, which require spatio-temporal reasoning from videos (e.g.(a) repetition count, (b) repeating action, and (c) state transition). It also includes frame QA tasks that can be answered from one of frames.
• Figure 3: The proposed ST-VQA model for spatio-temporal VQA. See Figure \ref{['fig:attention_diagram']} for the structure of spatial and temporal attention modules.
• Figure 4: Our spatial and temporal attention mechanisms.
• Figure 5: Qualitative comparison of VQA results from different approaches, on the four task types of our TGIF-QA dataset.