Consistent Video Colorization via Palette Guidance

TL;DR

This work tackles the dual challenges of color saturation and temporal inconsistency in video colorization by re-purposing Stable Video Diffusion (SVD) as a backbone and introducing a global, palette-based conditioning mechanism. A five-color palette is extracted and projected to a dense embedding $C_{emb}$ that is fused into the first encoder layer, with training optimized by a diffusion-based colorization loss $L$ and guidance from an image embedding $I_{emb}$; palettes can be derived from reference images, Gaussian mixture sampling, or large-language-model–driven color generation. The contributions include (1) an integrated diffusion-based colorization framework with global palette guidance, (2) flexible palette generation methods for diverse styles, and (3) substantial improvements in color saturation and temporal stability on the DAVIS2017 benchmark. This approach enables vivid, coherent colorization suitable for archival restoration and applications requiring stylistically controllable video colorization, without requiring ground-truth color palettes for each sequence.

Abstract

Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.

Figures (5)

• Figure 1: Comparison with frame independent colorization framework TCVCLiu2024Temporally. Our method shows superiority in color vividness and temporal consistence.
• Figure 2: Overview of the video colorization pipeline with palette-guidance. We re-purpose a pre-trained video diffusion model and augment it with palette guidance. The left side illustrates three methods for generating palettes. During training, the palette is generated in the manner depicted in (a), with the reference image being a randomly selected frame from each training clip.
• Figure 3: Visual comparison of colorization results for 2 videos. From left to right, the colorization results for Deoldifyantic_deoldify_nodate, VCGANzhou2023vcgan, TCVCLiu2024Temporally, and ours.
• Figure 4: Visual effects of ablation studies. The first column (NULL) shows the colorization results without palette guidance. The other columns display the colorization results of model variants with different palette generation methods.
• Figure 5: Diverse colorization results with distinct palettes.