Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos

TL;DR

This work introduces PhysGame, a large-scale glitch-based instruction-tuning dataset, and GameBench, an expert-annotated glitch benchmark, to advance physical world understanding in multimodal models. By extracting QA pairs from glitchy gameplay videos and guiding QA generation with meta-information, PhysGame enables scalable supervision across five physical domains and sixteen categories. Empirical results show that PhysGame improves real-world physical reasoning (Game2Real) and general video understanding (Game2General), with notable gains across multiple models and robust ablations highlighting the benefits of data scale, prompting strategies, and data mixing. The approach demonstrates that learning from gameplay anomalies can bridge sim-to-real gaps and enhance robustness in physical commonsense reasoning for multimodal AI systems.

Abstract

Understanding the physical world, including object dynamics, material properties, and causal interactions, remains a core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general reasoning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. In this paper, we propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding. We introduce PhysGame, an meta information guided instruction-tuning dataset containing 140,057 glitch-centric question-answer pairs across five physical domains and sixteen fine-grained categories. To ensure data accuracy, we design a prompting strategy that utilizes gameplay metadata such as titles and descriptions to guide high-quality QA generation. Complementing PhysGame, we construct GameBench, an expert-annotated benchmark with 880 glitch-identified gameplay videos designed to evaluate physical reasoning capabilities. Extensive experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real world physical reasoning performance of Qwen2.5VL by 2.5% on PhysBench, and Game2General transferability, yielding a 1.9% gain on the MVBench benchmark. Moreover, PhysGame-tuned models achieve a 3.7% absolute improvement on GameBench, demonstrating enhanced robustness in detecting physical implausibilities. These results indicate that learning from gameplay anomalies offers a scalable and effective pathway toward advancing physical world understanding in multimodal intelligence.

Figures (7)

• Figure 1: We propose PhysGame, an instruction-tuning dataset constructed from gameplay videos with glitch-induced question-answer pairs. Our method demonstrates consistent performance improvements on downstream benchmarks for both real-world physical understanding (Game2Real) and general video understanding (Game2General) tasks.
• Figure 2: Sampled examples in the proposed PhysGame instructing dataset (left) and the GameBench evaluation benchmark (right).
• Figure 3: (a) The taxonomy and (b) key statistics of the PhysGame and GameBench dataset.
• Figure 4: The generated responses in PhysGame dataset with or without meta-information hints (i.e., titles). The title indicates a glitch related to water physics. With the title hints, the generated responses accurately identify that the surface of the water lacks realistic wave propagation; in contrast, the response contains hallucinated content without the title hints.
• Figure 5: Data-scaling effects of our PhysGame dataset.