NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions

TL;DR

NExT-QA presents a large, manually annotated VideoQA benchmark focused on explaining temporal actions through causal and temporal reasoning, evaluated via both multi-choice and open-ended tasks. The dataset (5440 videos, ~52K QA pairs) enables diagnostic analyses across question types and supports rigorous baselines and state-of-the-art models with detailed ablations on video sampling and representations. Findings show that current methods perform well on descriptive questions but struggle with causal/temporal reasoning, with open-ended QA revealing even larger gaps and highlighting the need for more grounded, relational video understanding. The work advocates graph-based reasoning and context-sensitive language grounding as promising avenues and positions NExT-QA as a benchmark to drive the next generation of video reasoning research.

Abstract

We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial scene description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git)

Figures (13)

• Figure 1: NExT-QA is a question answering benchmark targeting the explanation of video contents. It challenges QA models to reason about causal and temporal actions and understand the rich object interactions in daily activities.
• Figure 2: Examples of multi-choice QA.
• Figure 3: Data statistics. (a) Distribution of the question types. (b) The average question length is 11.6, and the specific lengths for causal, temporal and descriptive questions are 12.1, 13.4 and 8.0 respectively. (c) The average answer length is 2.6. Specific lengths for causal, temporal and descriptive answers are 3, 2.8 and 1.4 respectively.
• Figure 4: (a) Results with different number of clips. (b) Results with different video representations. C, T and D stand for causal, temporal and descriptive questions respectively.
• Figure 5: Visualization of answer prediction results. For multi-choice QA, the correct answers and predictions are highlighted in red. For open-ended QA, the WUPS score of each prediction is appended. 'null' means the methods fail to generate any effective words. (C: Causal. T: Temporal. D: Descriptive.)