ColorizeDiffusion v2: Enhancing Reference-based Sketch Colorization Through Separating Utilities

TL;DR

The paper addresses spatial misalignment between sketches and references in reference-based sketch colorization, which leads to artifacts. It introduces a carrier-separation framework that combines image embeddings and latent features via split cross-attention with foreground/background masks, plus dedicated background and style encoders to improve spatial control and detail transfer. It also proposes preprocessing steps (character-mask merging and background bleaching) and a step-by-step training strategy to mitigate spatial entanglement and optimize each component. Evaluations on a large anime-style dataset show state-of-the-art colorization with reduced artifacts, stronger style transfer, and favorable user study results, with ablations validating each component.

Abstract

Reference-based sketch colorization methods have garnered significant attention due to their potential applications in the animation production industry. However, most existing methods are trained with image triplets of sketch, reference, and ground truth that are semantically and spatially well-aligned, while real-world references and sketches often exhibit substantial misalignment. This mismatch in data distribution between training and inference leads to overfitting, consequently resulting in spatial artifacts and significant degradation in overall colorization quality, limiting potential applications of current methods for general purposes. To address this limitation, we conduct an in-depth analysis of the \textbf{carrier}, defined as the latent representation facilitating information transfer from reference to sketch. Based on this analysis, we propose a novel workflow that dynamically adapts the carrier to optimize distinct aspects of colorization. Specifically, for spatially misaligned artifacts, we introduce a split cross-attention mechanism with spatial masks, enabling region-specific reference injection within the diffusion process. To mitigate semantic neglect of sketches, we employ dedicated background and style encoders to transfer detailed reference information in the latent feature space, achieving enhanced spatial control and richer detail synthesis. Furthermore, we propose character-mask merging and background bleaching as preprocessing steps to improve foreground-background integration and background generation. Extensive qualitative and quantitative evaluations, including a user study, demonstrate the superior performance of our proposed method compared to existing approaches. An ablation study further validates the efficacy of each proposed component.

Figures (21)

• Figure 1: We propose a novel reference-based sketch colorization framework that separates the colorization into different aspects and enhances them with respective modules. This system can achieve state-of-the-art anime-style colorization results without requiring spatial correspondence between inputs.
• Figure 2: An example of distribution shift in training and inference data. Existing reference-based methods typically derive references from ground truth with deformation due to the data limitation, the augmented references are still highly semantically related to the sketches. During inference, however, the reference images could be non-relevant to the sketches. This distribution gap between train and inference data results in overfitting, severely detereiorate the colorization quality.
• Figure 3: Spatial entanglement represented in different ways. The T2I method mistakenly changes hairstyle in the results. For the I2I methods, spatial entanglements are influenced by the carrier used for the color reference injection. Carriers containing less detail and higher-level semantic information tend to produce results with fewer artifacts and blurry textures, and vice versa. Source of results: 1. our vanilla mode; 2. fine-tuned IP-Adapter; 3. jointly-trained reference net.
• Figure 4: Illustration of the proposed framework with extracted carriers labeled by orange rectangles next to their corresponding encoder. We use masks to separate reference images into foreground and background for CLIP Image encoder $\phi$ to extract respective embeddings. Their concatenation is K and V inputs for split cross-attention in the denoising U-Net, and background embeddings are KV inputs for the background U-Net encoder. Note that we directly use ground truth color images as references in all the training mentioned in this paper. Sketch and mask images are generated from original color images using existing extraction methods sketchKerasanime-segmentation.
• Figure 5: Illustration of a decoder block in the denoising U-Net after merging the split cross-attention, the background encoder, and the style encoder. Note that only the LoRA weights in the split cross-attention are optimized in the proposed training.