Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering

TL;DR

The paper addresses the lack of interpretability and robust compositional reasoning in Video Question-Answering by introducing VA^3, a model-agnostic framework that combines a hierarchical video aligner with an answer aggregator operating over a Question Decomposition Graph (QDG). It also includes an automatic question-decomposition pipeline based on large language models to apply the framework to datasets lacking explicit QDGs, and proposes symmetric compositional metrics (cP, cR, c-F1 and their negative variants) to better capture reasoning consistency. Empirical results across AGQA-Decomp and extended datasets (MSVD, NExT-QA) show improvements in both main/sub-question accuracy and compositional consistency (notably c-F1) across multiple baselines, with ablations and qualitative analyses illustrating enhanced interpretability. Overall, VA^3 offers a practical, interpretable approach to improve compositional reasoning in VidQA while remaining compatible with existing backbones.

Abstract

Despite the recent progress made in Video Question-Answering (VideoQA), these methods typically function as black-boxes, making it difficult to understand their reasoning processes and perform consistent compositional reasoning. To address these challenges, we propose a \textit{model-agnostic} Video Alignment and Answer Aggregation (VA$^{3}$) framework, which is capable of enhancing both compositional consistency and accuracy of existing VidQA methods by integrating video aligner and answer aggregator modules. The video aligner hierarchically selects the relevant video clips based on the question, while the answer aggregator deduces the answer to the question based on its sub-questions, with compositional consistency ensured by the information flow along question decomposition graph and the contrastive learning strategy. We evaluate our framework on three settings of the AGQA-Decomp dataset with three baseline methods, and propose new metrics to measure the compositional consistency of VidQA methods more comprehensively. Moreover, we propose a large language model (LLM) based automatic question decomposition pipeline to apply our framework to any VidQA dataset. We extend MSVD and NExT-QA datasets with it to evaluate our VA$^3$ framework on broader scenarios. Extensive experiments show that our framework improves both compositional consistency and accuracy of existing methods, leading to more interpretable real-world VidQA models.

Figures (4)

• Figure 1: The Question Decomposition Graph (QDG) of a question from AGQA-Decomp gandhi2022measuring. The predicted answer to each question is from HQGA XiaoYL0JC22. Green (resp., Red) represents the predicted answer is right (resp., wrong)
• Figure 2: Our model-agnostic Video Alignment and Answer Aggregation (VA$^3$) framework. $(v, q)$ is a video-question pair, where $q_m$ denotes main-question and ${q_{s_1}, \cdots, q_{s_n}}$ denote the $n$ sub-questions derived from $q_m$. $v_m$ and $\{v_{s_1}, \cdots, v_{s_n}\}$ denote the aligned videos according to corresponding questions. $\bm{f}_{v_m, q_m}$ and $\{\bm{f}_{v_{s_1}, q_{s_1}}, \cdots, \bm{f}_{v_{s_n}, q_{s_n}}\}$ denote the video-question joint features. $G_{q_m}$ is the question decomposition graph (QDG) associated with $q_m$, which is a direct acyclic graph describing the compositional relationship among questions. Moreover, $G_{q_m}$ stores in which manner the questions are decomposited (i.e., the operators in the decomposition program) as the attribute of edges. $\hat{a}_m$ denotes the predicted answer for $q_m$.
• Figure 3: The automatic question decomposition pipeline. The question to be decomposed is denoted as $<$Question$>$.
• Figure 4: Quantitive results of Video Aligner and the visualization of improvements on accuracy and compositional consistency brought by our modules. Best viewed in color and zoom in. More visualizations and explanations are in the supplementary material.

Theorems & Definitions (2)

• Definition 1: Consistency Recalls
• Definition 2: Consistency F-Score