Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement

TL;DR

The paper tackles the misalignment between state-of-the-art video generation and real-world physics. It proposes a training-free, plug-and-play framework that uses LLMs and VLMs to iteratively refine prompts via MM-CoT to elicit physics-aware generation from VGMs. Empirical results on the PhyIQ benchmark show a notable increase in Physics-IQ from 56.31 to 62.38, with ensemble strategies further stabilizing outputs. The approach provides a blueprint for incorporating physical priors into video synthesis and highlights the potential of cross-modal prompting for physics-consistent content.

Abstract

Recent progress in video generation has led to impressive visual quality, yet current models still struggle to produce results that align with real-world physical principles. To this end, we propose an iterative self-refinement framework that leverages large language models and vision-language models to provide physics-aware guidance for video generation. Specifically, we introduce a multimodal chain-of-thought (MM-CoT) process that refines prompts based on feedback from physical inconsistencies, progressively enhancing generation quality. This method is training-free and plug-and-play, making it readily applicable to a wide range of video generation models. Experiments on the PhyIQ benchmark show that our method improves the Physics-IQ score from 56.31 to 62.38. We hope this work serves as a preliminary exploration of physics-consistent video generation and may offer insights for future research.

Figures (2)

• Figure 1: Overview of our method.
• Figure 2: Visualization of generated videos.