ReasVQA: Advancing VideoQA with Imperfect Reasoning Process

TL;DR

ReasVQA addresses a core challenge in VideoQA: leveraging imperfect, automatically generated reasoning to improve question answering over video data. It introduces a three-phase pipeline—Reasoning Generation, Reasoning Refinement, and Learning from Reasoning—and a multi-task training objective that combines QA and reasoning signals, with loss terms $\mathcal{L}_{st}$ and $\mathcal{L}_{mt}$ governed by $\alpha$ and $\beta$ where $0<\alpha,\beta<1$ and $\alpha+\beta=1$. Across three benchmarks (NExT-QA, STAR, IntentQA), ReasVQA achieves new state-of-the-art totals (+2.9, +7.3, +5.9 respectively) and demonstrates robustness to backbones such as InternVL and BLIP-FlanT5, validating the value of refining imperfect reasoning data for supervision. The work provides extensive analyses on the benefits of reasoning refinement, the importance of multi-task supervision, and the method’s applicability to complex reasoning tasks, offering a practical strategy to integrate intermediate reasoning into multimodal VideoQA systems. Overall, ReasVQA establishes a new paradigm for enhancing VideoQA by distilling structured reasoning from large multimodal models without leaking information during evaluation.

Abstract

Video Question Answering (VideoQA) is a challenging task that requires understanding complex visual and temporal relationships within videos to answer questions accurately. In this work, we introduce \textbf{ReasVQA} (Reasoning-enhanced Video Question Answering), a novel approach that leverages reasoning processes generated by Multimodal Large Language Models (MLLMs) to improve the performance of VideoQA models. Our approach consists of three phases: reasoning generation, reasoning refinement, and learning from reasoning. First, we generate detailed reasoning processes using additional MLLMs, and second refine them via a filtering step to ensure data quality. Finally, we use the reasoning data, which might be in an imperfect form, to guide the VideoQA model via multi-task learning, on how to interpret and answer questions based on a given video. We evaluate ReasVQA on three popular benchmarks, and our results establish new state-of-the-art performance with significant improvements of +2.9 on NExT-QA, +7.3 on STAR, and +5.9 on IntentQA. Our findings demonstrate the supervising benefits of integrating reasoning processes into VideoQA. Further studies validate each component of our method, also with different backbones and MLLMs, and again highlight the advantages of this simple but effective method. We offer a new perspective on enhancing VideoQA performance by utilizing advanced reasoning techniques, setting a new benchmark in this research field.

Figures (4)

• Figure 1: Overview of our method ReasVQA. ① Reasoning Generation: a SOTA MLLM is prompted to solve complex questions by generating detailed reasoning explanations. ② Reasoning Refinement: we process and refine the reasoning steps to alleviate the conflict with true answers. ③Learning from Reasoning (Multi-task Training): the refined reasoning steps are used to guide a model to improve its performance on the VideoQA tasks.
• Figure 2: An example of a response generated by an MLLM: the predicted final answer is "C", while the true answer is "A: The blanket". Although the final answer is incorrect, some of the reasoning steps still offer valuable learning elements for the model. For instance, sentences highlighted in green provide a partial description of the video and eliminate the possibility of two other options, providing meaningful insights.
• Figure 3: The impact of data quantity on performance and the comparison of Original and Refined Reasoning. Experiments are conducted using the multi-task learning approach, with $\alpha = \beta = 0.5$.
• Figure 4: Hyper-parameter tuning for the weight $\beta$ of the Reasoning Generation Loss $\mathcal{C}_{rea}(\cdot)$, with the corresponding $\alpha = 1 - \beta$.