PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos

TL;DR

This work introduces PhysGame, a comprehensive benchmark of 880 glitchy gameplay videos across four physical domains to evaluate physical commonsense in video LLMs. It provides two auxiliary datasets, PhysInstruct for instruction tuning and PhysDPO for preference optimization, and proposes PhysVLM, an open-source, physics-aware video LLM built on Qwen-7B via the PPLLaVA framework. Through extensive experiments, the authors show substantial gaps between open-source and proprietary models and demonstrate that PhysVLM achieves state-of-the-art performance on PhysGame and competitive results on general video benchmarks. The work also details ablations and prompts to guide future research in physical commonsense reasoning for video understanding.

Abstract

Recent advancements in video-based large language models (Video LLMs) have witnessed the emergence of diverse capabilities to reason and interpret dynamic visual content. Among them, gameplay videos stand out as a distinctive data source, often containing glitches that defy physics commonsense. This characteristic renders them an effective benchmark for assessing the under-explored capability of physical commonsense understanding in video LLMs. In this paper, we propose PhysGame as a pioneering benchmark to evaluate physical commonsense violations in gameplay videos. PhysGame comprises 880 videos associated with glitches spanning four fundamental domains (i.e., mechanics, kinematics, optics, and material properties) and across 12 distinct physical commonsense. Through extensively evaluating various state-ofthe-art video LLMs, our findings reveal that the performance of current open-source video LLMs significantly lags behind that of proprietary counterparts. To bridge this gap, we curate an instruction tuning dataset PhysInstruct with 140,057 question-answering pairs to facilitate physical commonsense learning. In addition, we also propose a preference optimization dataset PhysDPO with 34,358 training pairs, where the dis-preferred responses are generated conditioned on misleading titles (i.e., meta information hacking), fewer frames (i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking). Based on the suite of datasets, we propose PhysVLM as a physical knowledge-enhanced video LLM. Extensive experiments on both physical-oriented benchmark PhysGame and general video understanding benchmarks demonstrate the state-ofthe-art performance of PhysVLM.

Figures (6)

• Figure 1: The annotated multi-choice question in PhysGame. The correct option is annotated in green.
• Figure 2: Overview of the direct preference optimization training, where the preferred data is generated with the guidance of associated meta-information (i.e., title) while dispreferred data is generated with misleading titles (i.e., meta-information hacking), fewer frames (i.e., temporal hacking) and lower resolutions (i.e., spatial hacking).
• Figure 3: Example cases in the PhysInstruct dataset with (w/) or without (w/o) meta-information hints.
• Figure 4: Qualitative examples of open-ended questions.
• Figure 5: Qualitative examples of open-ended questions.