Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

TL;DR

This work tackles VideoQA by addressing the inadequacy of uniform frame sampling in Large Multimodal Models (LMMs) and the lack of human-annotated question-critical timestamps. It introduces a weakly supervised Gaussian-based Contrastive Grounding (GCG) framework that uses QA-derived event descriptions to generate pseudo-labels for temporal grounding via CLIP, learns multiple Gaussian masks to model temporal structure, and applies a differentiable Top-K selection to choose question-relevant frames. The model optimizes a joint objective comprising $\\mathcal{L}_{vqa}$, $\\mathcal{L}_{reg}$, and $\\mathcal{L}_{con}$, enabling end-to-end training without heavy video-text pretraining and yielding improved performance across six VideoQA benchmarks, especially on longer videos requiring causal-temporal reasoning. The approach also provides interpretable grounding of the selected moments, with potential implications for more efficient and reliable multimodal reasoning in video tasks, while acknowledging language-bias limitations and proposing future work to mitigate such biases.

Abstract

Video Question Answering (VideoQA) aims to answer natural language questions based on the information observed in videos. Despite the recent success of Large Multimodal Models (LMMs) in image-language understanding and reasoning, they deal with VideoQA insufficiently, by simply taking uniformly sampled frames as visual inputs, which ignores question-relevant visual clues. Moreover, there are no human annotations for question-critical timestamps in existing VideoQA datasets. In light of this, we propose a novel weakly supervised framework to enforce the LMMs to reason out the answers with question-critical moments as visual inputs. Specifically, we first fuse the question and answer pairs as event descriptions to find multiple keyframes as target moments and pseudo-labels, with the visual-language alignment capability of the CLIP models. With these pseudo-labeled keyframes as additionally weak supervision, we devise a lightweight Gaussian-based Contrastive Grounding (GCG) module. GCG learns multiple Gaussian functions to characterize the temporal structure of the video, and sample question-critical frames as positive moments to be the visual inputs of LMMs. Extensive experiments on several benchmarks verify the effectiveness of our framework, and we achieve substantial improvements compared to previous state-of-the-art methods.

Figures (7)

• Figure 1: The information in uniformly sampled frames is insufficient to answer the question correctly. We utilize the fused event description to provide weak supervision and generate weight distributions for each video moment. We align the positive description-moment pairs while pushing away negative ones.
• Figure 2: (a) The overall framework of our method. (b) The process of pseudo-label generation.
• Figure 3: We use the Gaussian generator to generate multiple Gaussian masks and obtain weight distributions $p\in \mathbb{R}^{T}$ for each video moment. The Gaussian generator will be optimized by the regression objective $\mathcal{L}_{reg}$ and contrastive objective $\mathcal{L}_{con}$, along with the fully supervised QA objective $\mathcal{L}_{vqa}$, to discover the most question-critical moments as visual inputs for LMMs.
• Figure 4: As a preliminary step, we analyze the performance upper bound with weakly labeled keyframes as visual inputs.
• Figure 5: Ablation studies on hyperparameters $T$ and $\sigma$.