LVCD: Reference-based Lineart Video Colorization with Diffusion Models

TL;DR

This work proposes the first video diffusion framework for reference-based lineart video colorization, and introduces Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart.

Abstract

We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at https://luckyhzt.github.io/lvcd.

Figures (9)

• Figure 1: Model architecture of sketch-guided ControlNet and Reference Attention. All frames are from Big Fish & Begonia.
• Figure 2: Sequential sampling with Overlapped Blending and Prev-Reference Attention.
• Figure 3: Qualitative comparison with five methods: ACOF flow_anime_color, TCVC temporal_video_color, CNet+Refonly controlnet, EISAI eisai, and SEINE seine. It is recommended to zoom in on the figure to observe the differences. Input frames: $1^{st}$ and $2^{nd}$ examples are from Big Fish & Begonia, $3^{rd}$ example is from Mr. Miao.
• Figure 4: Results of user study. Our method has a preferred rate of 58.3% (62.0%, 52.4%, and 63.2% for user group CG & CV, Art & Design, and others).
• Figure 5: Ablation study on model architecture and sampling scheme. All input frames are from Big Fish & Begonia.