Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation

Changgu Chen, Junwei Shu, Gaoqi He, Changbo Wang, Yang Li

TL;DR

Motion-Zero introduces a training-free, plug-and-play framework for zero-shot moving-object trajectory control in diffusion-based videos. It combines an Initial Noise Prior Module to seed position-aware latents, Spatial Constraints using cross-attention losses to enforce box containment, and a Shift Temporal Attention Mechanism to maintain temporal coherence, enabling arbitrary bounding-box trajectories across pre-trained models. Extensive qualitative and quantitative evaluations demonstrate improved trajectory control without sacrificing video quality, outperforming several state-of-the-art baselines on multiple metrics and in user studies. The approach offers practical, model-agnostic control for conditional video generation with broad applicability and minimal computational overhead during inference.

Abstract

Recent large-scale pre-trained diffusion models have demonstrated a powerful generative ability to produce high-quality videos from detailed text descriptions. However, exerting control over the motion of objects in videos generated by any video diffusion model is a challenging problem. In this paper, we propose a novel zero-shot moving object trajectory control framework, Motion-Zero, to enable a bounding-box-trajectories-controlled text-to-video diffusion model. To this end, an initial noise prior module is designed to provide a position-based prior to improve the stability of the appearance of the moving object and the accuracy of position. In addition, based on the attention map of the U-net, spatial constraints are directly applied to the denoising process of diffusion models, which further ensures the positional and spatial consistency of moving objects during the inference. Furthermore, temporal consistency is guaranteed with a proposed shift temporal attention mechanism. Our method can be flexibly applied to various state-of-the-art video diffusion models without any training process. Extensive experiments demonstrate our proposed method can control the motion trajectories of objects and generate high-quality videos. Our project page is https://vpx-ecnu.github.io/MotionZero-website/

Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation

TL;DR

Motion-Zero introduces a training-free, plug-and-play framework for zero-shot moving-object trajectory control in diffusion-based videos. It combines an Initial Noise Prior Module to seed position-aware latents, Spatial Constraints using cross-attention losses to enforce box containment, and a Shift Temporal Attention Mechanism to maintain temporal coherence, enabling arbitrary bounding-box trajectories across pre-trained models. Extensive qualitative and quantitative evaluations demonstrate improved trajectory control without sacrificing video quality, outperforming several state-of-the-art baselines on multiple metrics and in user studies. The approach offers practical, model-agnostic control for conditional video generation with broad applicability and minimal computational overhead during inference.

Abstract

Recent large-scale pre-trained diffusion models have demonstrated a powerful generative ability to produce high-quality videos from detailed text descriptions. However, exerting control over the motion of objects in videos generated by any video diffusion model is a challenging problem. In this paper, we propose a novel zero-shot moving object trajectory control framework, Motion-Zero, to enable a bounding-box-trajectories-controlled text-to-video diffusion model. To this end, an initial noise prior module is designed to provide a position-based prior to improve the stability of the appearance of the moving object and the accuracy of position. In addition, based on the attention map of the U-net, spatial constraints are directly applied to the denoising process of diffusion models, which further ensures the positional and spatial consistency of moving objects during the inference. Furthermore, temporal consistency is guaranteed with a proposed shift temporal attention mechanism. Our method can be flexibly applied to various state-of-the-art video diffusion models without any training process. Extensive experiments demonstrate our proposed method can control the motion trajectories of objects and generate high-quality videos. Our project page is https://vpx-ecnu.github.io/MotionZero-website/
Paper Structure (28 sections, 14 equations, 14 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 14 equations, 14 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Methodology
  5. Overview
  6. Initial Noise Prior Module
  7. Spatial Constraints with Attention Map
  8. Shift Temporal Attention Mechanism
  9. Experiments
  10. Implementation Details
  11. Comparisons with SOTA Methods
  12. Qualitative Analysis.
  13. Quantitative Analysis.
  14. Ablation Study
  15. User Study
  16. ...and 13 more sections

Figures (14)

  • Figure 1: Our Motion-Zero framework endows different pre-trained video diffusion models with the capability to manipulate object trajectories directly, circumventing the need for supplementary training. By designating the target entity in the input prompts and a sequence of bounding boxes, users can intuitively direct the motion path of the object within the generated video.
  • Figure 2: Overview of our Motion-Zero. The total pipeline is shown in (a). Given the box condition $\mathcal{B}$ and the prompt condition, we generate the prior latents $\mathbf{z}_T$ by our Initial Noise Prior Module (INPM) as shown on (b). At timestep $t$, $\mathbf{z}_t$ is firstly optimized to $\mathbf{z}_t'$ by the Spatial Constraints (SC). Subsequently, $\mathbf{z}_t'$ is passed to the UNet with Shift Temporal Attention Module (STAM) as demonstrated on (c). All the parameters of the video diffusion are frozen. $T_1$ represents the number of timesteps during which SC and STAM are applied, and $T_2$ denotes the number of timesteps where the original video diffusion process is utilized.
  • Figure 3: Quality comparison results on different methods. We take one frame from every three frames. The input prompt: A fish is swimming in the sea. We employed ModelScope (a) and ZeroScope (b) as our baseline models and compared the effect of incorporating additional prompts with the integration of our Motion-Zero. In addition, we conducted a comparative analysis with TrailBlazer and Peekaboo.
  • Figure 4: Quality comparison results with complex trajectories. We take one frame from every three frames. The input prompt of the first row: A penguin standing on an iceberg. The second row: A rocket launching into space from a launchpad. The third row: A rabbit burrowing downwards into its warren.Zoom in for the best view.
  • Figure 5: Attention maps with different components. Prompt: A seal walking on the ice.
  • ...and 9 more figures