Characterizing Motion Encoding in Video Diffusion Timesteps

TL;DR

The paper tackles how motion is encoded in video diffusion timesteps and reveals a robust spatiotemporal structure: motion is primarily encoded in early denoising steps, while appearance is refined later. It introduces a prompt-tampering probe and a DDIM-based inversion to quantify the trade-off between appearance editing and motion preservation across timesteps, discovering a consistent motion-dominant regime and an appearance-dominant regime across architectures. Building on this, the authors propose a timestep-constrained one-shot motion customization framework that tunes only early temporal attention (via LoRA) and constrains inference to motion-dominant timesteps, achieving strong motion transfer without debiasing modules or extensive staged training. The approach yields state-of-the-art results on TGVE with a simplified pipeline and demonstrates robust behavior across multiple base architectures, suggesting practical, scalable mechanisms for spatiotemporal disentanglement in video diffusion. The work provides a principled, quantitative lens for motion control in video synthesis and offers a ready-to-integrate recipe for motion-centric editing in diffusion-based pipelines.

Abstract

Text-to-video diffusion models synthesize temporal motion and spatial appearance through iterative denoising, yet how motion is encoded across timesteps remains poorly understood. Practitioners often exploit the empirical heuristic that early timesteps mainly shape motion and layout while later ones refine appearance, but this behavior has not been systematically characterized. In this work, we proxy motion encoding in video diffusion timesteps by the trade-off between appearance editing and motion preservation induced when injecting new conditions over specified timestep ranges, and characterize this proxy through a large-scale quantitative study. This protocol allows us to factor motion from appearance by quantitatively mapping how they compete along the denoising trajectory. Across diverse architectures, we consistently identify an early, motion-dominant regime and a later, appearance-dominant regime, yielding an operational motion-appearance boundary in timestep space. Building on this characterization, we simplify current one-shot motion customization paradigm by restricting training and inference to the motion-dominant regime, achieving strong motion transfer without auxiliary debiasing modules or specialized objectives. Our analysis turns a widely used heuristic into a spatiotemporal disentanglement principle, and our timestep-constrained recipe can serve as ready integration into existing motion transfer and editing methods.

Figures (4)

• Figure 1: Spatiotemporal disentanglement in video diffusion models. Our finding reveals that motion is primarily encoded in the early denoising timesteps. Given a reference video (top) and its ground truth caption (blue), we perform DDIM inversion and then denoise with a new prompt that modifies only the subject (yellow). The resampled videos show different subject editing and motion preservation results by applying the original or new prompts at different timesteps.
• Figure 2: Subject editing and motion preservation quality of ModelScope, Latte and CogVideoX. Applying the new subject editing prompt in longer timesteps always leads to stronger new subject representation in the generated video. However, starting resampling with the new prompt at early timesteps significantly harms the motion preservation although it doesn't modify the motion description. The trade-off curves show the optimal timesteps to decompose spatial and temporal signals. This spatiotemporal property holds consistently across different model architectures.
• Figure 3: One-shot video motion customization via denoising timestep constraint. Leveraging our spatiotemporal disentanglement property, we train LoRAs at only early denoising timesteps to model the reference motion without appearance leakage. This single-stage fine-tuning approach achieves surpassing performance without any additional debiasing modules, stages or losses. This even works for base models with unified spatiotemporal attentions, where we add LoRA on the full spatiotemporal sequence and it is still prevented from overfitting on the reference appearance.
• Figure 4: Qualitative comparison of our motion disentanglement method to previous SOTAs. Our method faithfully replicates the motion of the reference video while also editing the subject and background with superior quality to other approaches. Without any additional spatial debiasing modules or stages, our method is stable and robust with minimal semantic discrepancy (e.g. the snow ground and hat-like reef by MotionDirector, and the extra wall texture and missing object by MotionClone).