Latent Temporal Discrepancy as Motion Prior: A Loss-Weighting Strategy for Dynamic Fidelity in T2V

TL;DR

This work tackles the challenge of maintaining temporal coherence and high-frequency motion in text-to-video diffusion by introducing Latent Temporal Discrepancy ($LTD$) as a motion prior. LTD computes frame-to-frame latent differences within a sliding window and reweights the diffusion loss to emphasize dynamic regions, all while avoiding external motion estimation. Implemented in a latent-diffusion framework with a 3D VAE, the method demonstrates stable training and improved motion fidelity on VBench and VMBench, outperforming several baselines by up to 3.58% in motion-related metrics while preserving static quality. The approach is shown to be effective through extensive quantitative comparisons, ablations, human evaluation, and qualitative visual analyses, offering a practical, plug-and-play enhancement for motion-aware T2V generation.

Abstract

Video generation models have achieved notable progress in static scenarios, yet their performance in motion video generation remains limited, with quality degrading under drastic dynamic changes. This is due to noise disrupting temporal coherence and increasing the difficulty of learning dynamic regions. {Unfortunately, existing diffusion models rely on static loss for all scenarios, constraining their ability to capture complex dynamics.} To address this issue, we introduce Latent Temporal Discrepancy (LTD) as a motion prior to guide loss weighting. LTD measures frame-to-frame variation in the latent space, assigning larger penalties to regions with higher discrepancy while maintaining regular optimization for stable regions. This motion-aware strategy stabilizes training and enables the model to better reconstruct high-frequency dynamics. Extensive experiments on the general benchmark VBench and the motion-focused VMBench show consistent gains, with our method outperforming strong baselines by 3.31% on VBench and 3.58% on VMBench, achieving significant improvements in motion quality.

Figures (5)

• Figure 1: Noise prediction and noise squared error $\left\| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c}) \right\|^2$, Latent Temporal Discrepancy $\mathrm{ln}(e+D)$, and Video $\mathbf{x}_\mathrm{video}$ visualization.
• Figure 2: Overview of our method. We use a 3D VAE for video encoding and decoding, and adopt DiT as the diffusion model. A motion prior is built from temporal discrepancies of latent features in a sliding window, which guides the LTD Loss to enhance learning of motion.
• Figure 3: Line plot of mean Latent Temporal Discrepancy, Wan2.1 MSE loss, and Ours MSE Loss across latent frames.
• Figure 4: Human Evaluation. The figure illustrates, for each participant, the preference ratios for Ours, Wan2.1 wan2.1, and Indistinguishable cases.
• Figure 5: Qualitative comparison of generated videos. Wan2.1 wan2.1 may generate counterfactual motion directions where the subject violates physical plausibility, while our method yields more reasonable outcomes.