Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models

TL;DR

Cinemo tackles the dual challenges of image-to-video consistency and prompt-driven motion control by reframing image animation as motion residual learning within a latent diffusion framework. It allows the model to predict motion residuals $M$ conditioned on the input image $z_1$ while quantifying motion intensity through an SSIM-based bucket, and stabilizes inference noise with a DCT-based refinement ${\epsilon'}$, thereby improving temporal coherence and alignment with textual prompts. The method demonstrates superior performance against state-of-the-art baselines on UCF-101 and MSR-VTT across multiple metrics, and enables practical extensions such as motion transfer and video editing. The approach offers a scalable, controllable path for open-domain I2V with strong detail preservation, though it currently relies on the backbone capabilities of LaVie and operates at a fixed resolution, suggesting directions toward Transformer-based architectures and higher-resolution training.

Abstract

Diffusion models have achieved great progress in image animation due to powerful generative capabilities. However, maintaining spatio-temporal consistency with detailed information from the input static image over time (e.g., style, background, and object of the input static image) and ensuring smoothness in animated video narratives guided by textual prompts still remains challenging. In this paper, we introduce Cinemo, a novel image animation approach towards achieving better motion controllability, as well as stronger temporal consistency and smoothness. In general, we propose three effective strategies at the training and inference stages of Cinemo to accomplish our goal. At the training stage, Cinemo focuses on learning the distribution of motion residuals, rather than directly predicting subsequent via a motion diffusion model. Additionally, a structural similarity index-based strategy is proposed to enable Cinemo to have better controllability of motion intensity. At the inference stage, a noise refinement technique based on discrete cosine transformation is introduced to mitigate sudden motion changes. Such three strategies enable Cinemo to produce highly consistent, smooth, and motion-controllable results. Compared to previous methods, Cinemo offers simpler and more precise user controllability. Extensive experiments against several state-of-the-art methods, including both commercial tools and research approaches, across multiple metrics, demonstrate the effectiveness and superiority of our proposed approach.

Figures (9)

• Figure 1: Explanations of image consistency and motion controllability. Frames in (b) and (c) are image animation results obtained from PIA zhang2023pia and SEINE chen2023seine, respectively. We use "windmill turning" as the text descriptions. (b) The frames show clear differences in color and texture. In (c), the entire house is moving, which does not match the information provided in the textual prompt.
• Figure 2: Model pipeline overview. During training, instead of predicting the subsequent frames directly, our model learns the distribution of motion residuals, while providing effective motion intensity control. The details of the training procedure can be seen in Algorithm. \ref{['alg:training']}. During inference, we use Discrete Cosine Transformation to extract low-frequency components to refine the inference noise, which can stabilize the generation process of image animation.
• Figure 3: Influence of the FFT and DCT decomposition. The prompt is "a robot dancing". Best viewed with Acrobat Reader. Please click the image to view the animated videos.
• Figure 4: Qualitative visual comparisons between the baselines and our model. We compare our approach with both closed-source commercial tools and research works. "Girl smiling" means the used prompt when the method accepts it. Best viewed with Acrobat Reader. Please click the image to view the animated videos.
• Figure 5: Effectiveness of DCTInit. The middle video is generated by our model without enabling DCTInit. The prompt is "woman smiling". Best viewed with Acrobat Reader. Please click the image to view the animated videos.