Physical Simulator In-the-Loop Video Generation

TL;DR

A novel framework that integrates a physical simulator into the video diffusion process, and a Test-Time Texture Consistency Optimization technique that adapts text and feature embeddings based on pixel correspondences from the simulator to improve texture consistency during object movement.

Abstract

Recent advances in diffusion-based video generation have achieved remarkable visual realism but still struggle to obey basic physical laws such as gravity, inertia, and collision. Generated objects often move inconsistently across frames, exhibit implausible dynamics, or violate physical constraints, limiting the realism and reliability of AI-generated videos. We address this gap by introducing Physical Simulator In-the-loop Video Generation (PSIVG), a novel framework that integrates a physical simulator into the video diffusion process. Starting from a template video generated by a pre-trained diffusion model, PSIVG reconstructs the 4D scene and foreground object meshes, initializes them within a physical simulator, and generates physically consistent trajectories. These simulated trajectories are then used to guide the video generator toward spatio-temporally physically coherent motion. To further improve texture consistency during object movement, we propose a Test-Time Texture Consistency Optimization (TTCO) technique that adapts text and feature embeddings based on pixel correspondences from the simulator. Comprehensive experiments demonstrate that PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity. Project Page: https://vcai.mpi-inf.mpg.de/projects/PSIVG/

Figures (6)

• Figure 1: A baseline video generator (left) produces a physically implausible bowling collision with chaotic motion vectors across the video frames. Our PSIVG framework (right) integrates a physical simulator into the generation loop, guiding the video generator to produce a physically plausible and temporally coherent video.
• Figure 2: Overview of our Physical Simulator In-the-loop Video Generation (PSIVG) framework. From an input prompt, a template video is first generated, and is processed by our perception pipeline (Sec. \ref{['sec:perceptionpipeline']}). The outputs of the perception pipeline are further processed before being passed into the physical simulator (Sec. \ref{['sec:physicalsimulation']}). The rendered outputs from the simulator are then used for video generation (Sec. \ref{['sec:videogen']}), and this video generation can be improved with TTCO (see Sec. \ref{['sec:testtimeopt']}) for better texture consistency.
• Figure 3: Visualization of the sub-steps in our perception.
• Figure 4: Overview of TTCO. To improve the consistencies of textures, during test time, we add learnable zero-initialized embeddings to the text prompt and features, and optimize them with the outputs from the physical simulator. This allows the generated video to adhere to the simulator trajectories and rotations better, thereby improving the texture consistency.
• Figure 5: Qualitative comparisons, showing a teddy bear being dropped (left) and objects colliding (right). See Supp. Mat. for more results.