MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models

TL;DR

This work addresses the mismatch between photorealism and physically plausible motion in text-to-video diffusion. It introduces MoAlign, a two-stage fine-tuning framework that (i) learns a disentangled motion subspace from a pretrained video encoder by predicting optical flow, and (ii) aligns diffusion model latent features to this subspace with soft relational alignment, avoiding external simulators or conditioning. The approach yields stronger temporal coherence and physical plausibility while preserving semantic fidelity across multiple benchmarks and a human study, highlighting the benefits of motion-centric latent-space alignment. MoAlign demonstrates that internalizing motion priors in the diffusion model can substantially improve motion realism without sacrificing general video quality or prompt adherence.

Abstract

Text-to-video diffusion models have enabled high-quality video synthesis, yet often fail to generate temporally coherent and physically plausible motion. A key reason is the models' insufficient understanding of complex motions that natural videos often entail. Recent works tackle this problem by aligning diffusion model features with those from pretrained video encoders. However, these encoders mix video appearance and dynamics into entangled features, limiting the benefit of such alignment. In this paper, we propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder. This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics. We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos. Our method improves the physical commonsense in a state-of-the-art video diffusion model, while preserving adherence to textual prompts, as evidenced by empirical evaluations on VideoPhy, VideoPhy2, VBench, and VBench-2.0, along with a user study.

Figures (4)

• Figure 1: Problem(Left): Physics laws are often violated in outputs of video diffusion models. Base CogVideoX model (top) cannot generate a coin falling into a cup: the coin floats in the air instead. Our MoAlign method (bottom) improves this. Proposed solution(Right): Our finetuning pipeline aligns internal representations of the diffusion model with motion-specific features extracted from VideoMAEv2.
• Figure 2: Overview of our motion-centric fine-tuning framework. Stage 1 trains a motion-aware teacher by extracting features from a pretrained video encoder and supervising them with ground-truth optical flow. Stage 2 aligns the latent features of the video diffusion model (MM-DiT) to the motion-specific subspace via a soft relational alignment loss. This two-stage process internalizes motion understanding without requiring external conditioning or simulation at inference time.
• Figure 3: Qualitative results. MoAlign shows both glasses and a coherent pouring motion, while baselines miss key elements or exhibit inconsistent fluid behavior.
• Figure A.1: Qualitative comparison across methods for three prompts. In the first video, our method (MoAlign) preserves realistic human motion without deformation. In the second, it captures accurate hand-mouth interaction while baselines fail to represent the subject. In the third, it models physically plausible ketchup flow, unlike the erratic behavior seen in baselines. See supplementary videos for full comparisons.