Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sketch-based Fluid Video Generation Using Motion-Guided Diffusion Models in Still Landscape Images

Hao Jin, Haoran Xie

TL;DR

The paper tackles animating fluid dynamics within still landscape images by introducing a two-stage framework: first, a motion-field generator converts hand-drawn sketches into colored motion fields using a finetuned conditional latent diffusion model; second, a motion adapter integrates these fields into a Latent Video Diffusion Model to produce temporally coherent, motion-guided fluid videos. A single, sketch-derived motion field guides the diffusion-based synthesis, leveraging the temporal transformer in a 3D U-Net to maintain consistency while reducing training cost. The approach supports both naturalistic and anime-style landscapes, and experiments show quantitative and qualitative improvements over Stable Video Diffusion, highlighting the method's effectiveness for creative video production and intuitive user control.

Abstract

Integrating motion into static images not only enhances visual expressiveness but also creates a sense of immersion and temporal depth, establishing it as a longstanding and impactful theme in artistic expression. Fluid elements such as waterfall, river, and oceans are common features in landscape, but their complex dynamic characteristics pose significant challenges in modeling and controlling their motion within visual computing. Physics-based methods are often used in fluid animation to track particle movement. However, they are easily affected by boundary conditions. Recently, latent diffusion models have been applied to video generation tasks, demonstrating impressive capabilities in producing high-quality and temporally coherent results. However, it is challenging for the existing methods to animate fluid smooth and temporally consistent motion. To solve these issues, this paper introduces a framework for generating landscape videos by animating fluid in still images under the guidance of motion sketches. We propose a finetuned conditional latent diffusion model for generating motion field from user-provided sketches, which are subsequently integrated into a latent video diffusion model via a motion adapter to precisely control the fluid movement.

Sketch-based Fluid Video Generation Using Motion-Guided Diffusion Models in Still Landscape Images

TL;DR

The paper tackles animating fluid dynamics within still landscape images by introducing a two-stage framework: first, a motion-field generator converts hand-drawn sketches into colored motion fields using a finetuned conditional latent diffusion model; second, a motion adapter integrates these fields into a Latent Video Diffusion Model to produce temporally coherent, motion-guided fluid videos. A single, sketch-derived motion field guides the diffusion-based synthesis, leveraging the temporal transformer in a 3D U-Net to maintain consistency while reducing training cost. The approach supports both naturalistic and anime-style landscapes, and experiments show quantitative and qualitative improvements over Stable Video Diffusion, highlighting the method's effectiveness for creative video production and intuitive user control.

Abstract

Integrating motion into static images not only enhances visual expressiveness but also creates a sense of immersion and temporal depth, establishing it as a longstanding and impactful theme in artistic expression. Fluid elements such as waterfall, river, and oceans are common features in landscape, but their complex dynamic characteristics pose significant challenges in modeling and controlling their motion within visual computing. Physics-based methods are often used in fluid animation to track particle movement. However, they are easily affected by boundary conditions. Recently, latent diffusion models have been applied to video generation tasks, demonstrating impressive capabilities in producing high-quality and temporally coherent results. However, it is challenging for the existing methods to animate fluid smooth and temporally consistent motion. To solve these issues, this paper introduces a framework for generating landscape videos by animating fluid in still images under the guidance of motion sketches. We propose a finetuned conditional latent diffusion model for generating motion field from user-provided sketches, which are subsequently integrated into a latent video diffusion model via a motion adapter to precisely control the fluid movement.
Paper Structure (4 sections, 3 figures, 1 table)

This paper contains 4 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Results
  4. Conclusion

Figures (3)

  • Figure 1: The workflow of our proposed sketch-based video fluid generation framework.
  • Figure 2: Comparison between the generated motion fields with the colored representation.
  • Figure 3: Examples of fluid video generation (c) guided by motion fields (b) produced from motion sketches (a). (These videos are embedded and better viewed using Adobe Reader.)