PhysVideoGenerator: Towards Physically Aware Video Generation via Latent Physics Guidance
Siddarth Nilol Kundur Satish, Devesh Jaiswal, Hongyu Chen, Abhishek Bakshi
TL;DR
PhysVideoGenerator addresses the gap between high-quality video generation and learned physical dynamics by injecting latent physics priors derived from a pre-trained predictive world-model (V-JEPA 2) into a Latte-based diffusion generator. The framework introduces PredictorP, a lightweight network that regresses physics tokens from noisy latents, and a temporal cross-attention mechanism that injects these tokens into the diffusion process, enabling physics-informed motion without at inference time external simulators. The paper demonstrates training feasibility and stability of a joint objective that couples diffusion denoising with physics prediction over 50 epochs on resource-constrained hardware, laying a foundation for large-scale evaluation and future refinements in efficiency and scalability. While full generative quality benchmarks remain for future work, the results establish that diffusion latents contain recoverable physical information and that physics-guided conditioning can be integrated in a memory-conscious, end-to-end trainable manner, with implications for physically plausible, long-horizon video generation.
Abstract
Current video generation models produce high-quality aesthetic videos but often struggle to learn representations of real-world physics dynamics, resulting in artifacts such as unnatural object collisions, inconsistent gravity, and temporal flickering. In this work, we propose PhysVideoGenerator, a proof-of-concept framework that explicitly embeds a learnable physics prior into the video generation process. We introduce a lightweight predictor network, PredictorP, which regresses high-level physical features extracted from a pre-trained Video Joint Embedding Predictive Architecture (V-JEPA 2) directly from noisy diffusion latents. These predicted physics tokens are injected into the temporal attention layers of a DiT-based generator (Latte) via a dedicated cross-attention mechanism. Our primary contribution is demonstrating the technical feasibility of this joint training paradigm: we show that diffusion latents contain sufficient information to recover V-JEPA 2 physical representations, and that multi-task optimization remains stable over training. This report documents the architectural design, technical challenges, and validation of training stability, establishing a foundation for future large-scale evaluation of physics-aware generative models.