Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

TL;DR

This work tackles the lack of physical realism in video diffusion by injecting partial 3D information through per-pixel 3D trajectories tracked in the first-frame foreground and aligned with RGB pixels. It introduces the PointVid dataset and a joint training framework that augments diffusion denoisers with a 3D modality via cross-attention, followed by a regularization stage that enforces reconstruction fidelity and local rigidity in 3D trajectories. The approach yields improved shape and motion coherence, reduces non-physical artifacts such as morphing, and enhances performance on task-oriented, contact-rich videos when finetuned on two base models. The methodology provides a practical, architecture-agnostic path to endow video diffusion models with 3D priors, potentially advancing realism in applications from entertainment to scientific visualization.

Abstract

We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.

Figures (9)

• Figure 1: Comparison on Task-Oriented Videos. We present videos generated by different baselines (CogVideoX 1.5 yang2024cogvideox, SVD blattmann2023stablevideodiffusionscaling, DynamiCrafter xing2025dynamicrafter, LTX Video hacohen2024ltx and I2VGen-XL zhang2023i2vgen) and compare them with our method. We use the same input conditions for all methods (except that SVD is conditioned only on the image). It can be observed that existing baselines often exhibit severe distortions of hands or objects during human hand-object interactions. In contrast, our method preserves the shapes of both the hand and object during such interactions.
• Figure 2: Method Overview. During training, we sample video-point pairs, concatenate them along the channel dimensions, and use the augmented data to train a latent diffusion model. We introduce cross-attention between video and point data in corresponding channels to enhance alignment between the two modalities. The model predicts both RGB video and 3D points, leveraging the 3D information to further regularize video generation by applying a misalignment penalty during the diffusion process. During inference, the model generates both video and points from random noise, conditioned on a text-image prompt.
• Figure 3: PointVid Dataset Generation. Given an input video, we use the first frame as a reference frame and perform semantic segmentation to obtain masks for foreground objects. Next, we randomly sample pixels with a distribution favoring pixels inside foreground objects. We perform 3D point tracking on these queried pixels, and map these points to the input video frames. The resulting data point contains 3D coordinates of tracked foreground pixels while remaining pixels are zeroed out.
• Figure 4: Point Regularization. The reconstructed point cloud in the diffusion output often contains noise and deformations (middle). This issue is mitigated using our point regularization (right). The synthetic point cloud above (e.g., box and shoes falling on the ground) is generated by Kubric kubric and trained with our pipeline.
• Figure 5: Qualitative Comparison. We compare our UNet and DiT models against their respective baselines. The results show that both base models exhibit unrealistic artifacts, such as morphing, while our models ensure smooth transitions in object shape and motion.