Moaw: Unleashing Motion Awareness for Video Diffusion Models

TL;DR

Moaw tackles controllable motion transfer in video diffusion models by introducing a motion-perception diffusion network that predicts a dense 3D trajectory $P \in \mathbb{R}^{T \times H \times W \times 4}$ from a video and then injects motion-sensitive features into a structurally identical video generator in a zero-shot fashion. The approach combines a two-stage pipeline, latent-space diffusion, and cross-attentive conditioning to bridge motion understanding with video synthesis, and employs a ControlNet-inspired feature injection strategy that uses features from the motion-perception network without adapters. Empirically, Moaw achieves competitive 2D optical-flow metrics while delivering state-of-the-art inference efficiency for 3D dense tracking and substantially faster motion transfer than baselines such as DAS, validating the practicality of adapter-free motion control in diffusion-based video models. The work offers a new paradigm for unifying generative modeling and motion understanding, with implications for more controllable and scalable video learning systems.

Abstract

Video diffusion models, trained on large-scale datasets, naturally capture correspondences of shared features across frames. Recent works have exploited this property for tasks such as optical flow prediction and tracking in a zero-shot setting. Motivated by these findings, we investigate whether supervised training can more fully harness the tracking capability of video diffusion models. To this end, we propose Moaw, a framework that unleashes motion awareness for video diffusion models and leverages it to facilitate motion transfer. Specifically, we train a diffusion model for motion perception, shifting its modality from image-to-video generation to video-to-dense-tracking. We then construct a motion-labeled dataset to identify features that encode the strongest motion information, and inject them into a structurally identical video generation model. Owing to the homogeneity between the two networks, these features can be naturally adapted in a zero-shot manner, enabling motion transfer without additional adapters. Our work provides a new paradigm for bridging generative modeling and motion understanding, paving the way for more unified and controllable video learning frameworks.

Figures (7)

• Figure 1: Moaw is a framework that incorporates a motion perception diffusion network and injects its features into a video generation diffusion model in a zero-shot manner to achieve motion transfer.
• Figure 2: Overall pipeline. Our pipeline consists of two stages. The first stage is motion perception from a reference video. We encode the reference video into the latent space using the SVD video encoder, then sample Gaussian noise and concatenate it with the reference video latent along the channel dimension. A U-Net is then employed to predict the clean latent (without noise), which is finally decoded into a 3D dense trajectory. The second stage is video generation. Here, we take the features extracted from the first stage and inject them into a fixed SVD U-Net. Conditioned on an input image, the model generates a new video whose motion resembles that of the reference video.
• Figure 3: Motion-labeled video dataset. We construct a motion-labeled video dataset using images from ScanNet++ and the Stable Virtual Camera model. For each image, we generate six motion sequences, with each video consisting of 20 frames.
• Figure 4: PCA analysis for our motion perception video diffusion model. We select four motion patterns, and for each pattern randomly sample ten videos, resulting in 40 samples per feature. We then apply PCA for dimensionality reduction and visualization of these features.
• Figure 5: Visualizations. It shows qualitative comparisons between model predictions and ground truth for two samples. The top two rows correspond to examples from the training set, while the bottom two rows are from the validation set. Each sample includes 10 subfigures: the first row shows model predictions, and the second row shows the corresponding ground truth. From left to right, the columns display: (1) visualizations of subsampled 2D dense trajectories, (2) predicted colored tracks, (3) trajectory depth, (4) visibility maps, and (5) visualizations of 3D trajectoy.