A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs

TL;DR

The paper tackles the problem of evaluating physical world understanding in video-language systems, which is confounded by shortcuts that exploit superficial cues. It introduces MVP, a large-scale, shortcut-aware video QA benchmark built from nine data sources, using minimally different video pairs and a minimal-pair scoring scheme to compel models to process temporal and physical cues rather than rely on language or visual biases. Empirical results reveal a stark gap between human performance (≈92.9%) and the best open- or closed-source systems (≈40.2%), underscoring the need for more robust benchmarks and training data for world-modelling. By analyzing MVBench robustness, detailing an automated minimal-change mining pipeline, and demonstrating the impact of bias-filtering steps, MVP provides a scalable framework to drive progress toward physically grounded, temporally aware video understanding.

Abstract

Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.

Figures (2)

• Figure 1: Illustrating MVP with its curation steps (top) and examples of our Minimal Pair Scoring (bottom).
• Figure 2: Performance of the strongest evaluated VideoLLMs on MVP (mini-version), compared to human performance.