ReVision: Refining Video Diffusion with Explicit 3D Motion Modeling

TL;DR

ReVision tackles the persistent challenge of generating videos with realistic, controllable motion by explicitly incorporating 3D motion knowledge into a pretrained video diffusion model. It introduces a three-stage pipeline that first produces a coarse video, then extracts and refines 3D motion via a Parameterized Motion Prior (PMP), and finally reconditions the diffusion model with the improved motion to achieve coherent, physically plausible videos. The method leverages 3D object-centric representations, including SMPL-X for humans, SMAL for animals, and 2.5D representations for general objects, to guide motion optimization, while a transformer-based PMP denoises and stabilizes motion sequences using text conditioning and motion strength cues. Empirical results on Stable Video Diffusion show substantial improvements in motion fidelity and interaction realism, often outperforming larger baselines, and demonstrating that integrating structured 3D priors can yield high-quality, motion-rich videos with smaller models and efficient inference.

Abstract

In recent years, video generation has seen significant advancements. However, challenges still persist in generating complex motions and interactions. To address these challenges, we introduce ReVision, a plug-and-play framework that explicitly integrates parameterized 3D model knowledge into a pretrained conditional video generation model, significantly enhancing its ability to generate high-quality videos with complex motion and interactions. Specifically, ReVision consists of three stages. First, a video diffusion model is used to generate a coarse video. Next, we extract a set of 2D and 3D features from the coarse video to construct a 3D object-centric representation, which is then refined by our proposed parameterized motion prior model to produce an accurate 3D motion sequence. Finally, this refined motion sequence is fed back into the same video diffusion model as additional conditioning, enabling the generation of motion-consistent videos, even in scenarios involving complex actions and interactions. We validate the effectiveness of our approach on Stable Video Diffusion, where ReVision significantly improves motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even outperforms a state-of-the-art video generation model with over 13B parameters on complex video generation by a substantial margin. Our results suggest that, by incorporating 3D motion knowledge, even a relatively small video diffusion model can generate complex motions and interactions with greater realism and controllability, offering a promising solution for physically plausible video generation.

Figures (11)

• Figure 1: By explicitly leveraging a parameterized 3D motion model, ReVision enhances pre-trained video generation models (e.g., Stable Video Diffusion) to produce high-quality videos with complex motion (row 1), enabling precise motion control (rows 2, 3) and accurate interactions (rows 4, 5). During inference, an optional target pose can be specified via a rough sketch (rows 1, 3, 4, colored circles for different parts, dashed lines for the original pose, solid lines for the target pose) or a simple drag operation (blue arrows in row 2) indicating the final position.
• Figure 2: Method overview. Given the video generation model, ReVision operates in three stages. Stage 1: A coarse video is generated based on the provided conditions (e.g., target pose, marked in blue, indicating the rough position of the yellow part in the last frame). Stage 2: 3D features from the generated coarse video are extracted and optimized using the proposed PMP. Stage 3: The optimized 3D sequences are used to regenerate the video with enhanced motion consistency. Best viewed when zoomed in.
• Figure 3: Motion-conditioned video generation. We enable motion-conditioned generation by introducing two extra conditioning channels: (1) part segmentation mask derived from the 3D motion sequence, and (2) its corresponding confidence map.
• Figure 4: Qualitative comparisons. ReVision generates high-quality videos with complex motions and interactions of humans, animals, and general objects. Zoom in for better details. Please find the side-by-side video comparisons in the supplementary video. Reference frames are in the first column.
• Figure 5: User preference comparisons. Our model enhances the motion generation capability of the pre-trained SVD. It even surpasses HunyuanVideo, a SOTA model with 13B parameters. These results highlight the effectiveness of our model in generating complex motions and interactions.