ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

TL;DR

ConCLVD tackles the challenge of generating dynamic Chinese landscape painting videos by coupling a frozen Stable Diffusion backbone with a trainable motion module and a noise-adapter that enables unsupervised contrastive learning in latent space. It introduces CLV-HD, a dataset of approximately 1,300 text-video clips, to capture the unique aesthetics of landscape ink-wash art. The method combines a dual-attention motion module (Versatile Attention and Sparse-Causal Attention) with frame interpolation via sparse optical-flow projection to produce coherent, stylistically faithful videos while maintaining computational efficiency. Empirical results demonstrate strong qualitative and quantitative performance, with favorable user study feedback and notable cost advantages over several baselines. This work advances artistic video generation by preserving traditional brushwork and ink techniques in a dynamic, accessible diffusion-based framework.

Abstract

Chinese landscape painting is a gem of Chinese cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic Chinese landscape paintings. However, challenges such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating Chinese landscape painting videos. In this paper, we propose CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. The source code and dataset are available at https://anonymous.4open.science/r/ConCLVD-EFE3.

Figures (8)

• Figure 1: Overview of ConCLVD. Left: the architecture. ConCLVD integrates a trainable motion module based on a frozen Stable Diffusion and introduces a noise adapter to accommodate contrast learning of noise in latent space. Right: the inference framework. The video is generated by the denoising U-Net integrated with the motion module and enhanced by sparse optical flow projection (SOFP) interpolation technology.
• Figure 2: Design of Motion Module. The motion module is inserted following each image layer of the pre-trained SD to process video data.
• Figure 3: Detailed explanation of the attention mechanism. The above represents Versatile Attention, where each frame is related to every other frame; the below represents Sparse-Causal Attention, where each frame only focuses on its previous frame.
• Figure 4: Illustration of of SOFP Interpolation. $V_{(t \rightarrow T)}$ is the optical flow from $I_t$ to $I_T$ and $V_{(t \rightarrow t+1)}$ is the optical flow from $I_t$ to $I_{(t+1)}$. The projection result of $V_{(t \rightarrow T)}$ onto $V_{(t \rightarrow t+1)}$ is used to insert a new frame $I_{(t+0.5)}$ between two consecutive frames $I_t$ and $I_{(t+1)}$.
• Figure 5: Main Results. Our ConCLVD creates high-quality videos in the style of Chinese landscape painting.