A Video Is Not Worth a Thousand Words

TL;DR

The paper investigates how video information is integrated in vision-language models for multi-modal VQA and questions the extent of cross-modal interaction. It introduces a joint attribution framework based on Shapley values that can quantify both feature-level (frames vs textual elements) and modality-level contributions across video, question, and answer. By benchmarking six models on four diverse VQA datasets, the authors find that video is generally under-utilized relative to text, and that adding more multiple-choice options can artificially raise task difficulty and increase the model's reliance on video. The framework provides a flexible, interpretable tool for diagnosing and guiding improvements in multi-modal understanding.

Abstract

As we become increasingly dependent on vision language models (VLMs) to answer questions about the world around us, there is a significant amount of research devoted to increasing both the difficulty of video question answering (VQA) datasets, and the context lengths of the models that they evaluate. The reliance on large language models as backbones has lead to concerns about potential text dominance, and the exploration of interactions between modalities is underdeveloped. How do we measure whether we're heading in the right direction, with the complexity that multi-modal models introduce? We propose a joint method of computing both feature attributions and modality scores based on Shapley values, where both the features and modalities are arbitrarily definable. Using these metrics, we compare $6$ VLM models of varying context lengths on $4$ representative datasets, focusing on multiple-choice VQA. In particular, we consider video frames and whole textual elements as equal features in the hierarchy, and the multiple-choice VQA task as an interaction between three modalities: video, question and answer. Our results demonstrate a dependence on text and show that the multiple-choice VQA task devolves into a model's ability to ignore distractors. Code available at https://github.com/sjpollard/a-video-is-not-worth-a-thousand-words.

Figures (21)

• Figure 1: Per-Feature Contribution and accuracy as new negative answers are injected into the VQA-tuples, varying from easiest to hardest.
• Figure 2: Matrix of Shapley values per subset, where each row, left-to-right, represents the features of a VQA-tuple. Rows are truncated to a maximum of $200$ features.
• Figure 3: Qualitative figure of an example from EgoSchema evaluated using VideoLLaMA3. For brevity, we select the $16$ most important frames, ranked by the magnitude of their Shapley values. Here blue represents positively attributed inputs whereas red represents negatively attributed inputs.
• Figure 4: Plot of the MSE (mean squared error) of Shapley values at varying iterations for FrozenBiLM on the longest EgoSchema question. Error is calculated against values for $10,000$ iterations.
• Figure 5: Violin plots of the Spearman's correlations between the Gemini rankings and the Shapley value rankings for the frames of VQA-tuple questions.

Theorems & Definitions (4)

• Definition 3.1: The Shapley value
• Definition 3.2: Additive feature attribution
• Definition B.1: Additive feature attribution
• Theorem B.1