Follow-Your-Color: Multi-Instance Sketch Colorization

TL;DR

Follow-Your-Color tackles the inefficiency and inconsistency of multi-instance sketch colorization by introducing a diffusion-based framework with self-play training, an instance control module, and edge-aware color matching. It leverages Latent Diffusion Models and a dual-branch reference conditioning scheme to achieve per-instance chromatic fidelity in a single forward pass. The method demonstrates superior qualitative and quantitative performance over prior work, enabling automated, stylistically coherent colorization with multiple references. The approach broadens applicability to diverse references and promises to streamline animation pipelines and digital art workflows.

Abstract

We present Follow-Your-Color, a diffusion-based framework for multi-instance sketch colorization. The production of multi-instance 2D line art colorization adheres to an industry-standard workflow, which consists of three crucial stages: the design of line art characters, the coloring of individual objects, and the refinement process. The artists are required to repeat the process of coloring each instance one by one, which is inaccurate and inefficient. Meanwhile, current generative methods fail to solve this task due to the challenge of multi-instance pair data collection. To tackle these challenges, we incorporate three technical designs to ensure precise character detail transcription and achieve multi-instance sketch colorization in a single forward pass. Specifically, we first propose the self-play training strategy to address the lack of training data. Then we introduce an instance guider to feed the color of the instance. To achieve accurate color matching, we present fine-grained color matching with edge loss to enhance visual quality. Equipped with the proposed modules, Follow-Your-Color enables automatically transforming sketches into vividly-colored images with accurate consistency and multi-instance control. Experiments on our collected datasets show that our model outperforms existing methods regarding chromatic precision. Specifically, our model critically automates the colorization process with zero manual adjustments, so novice users can produce stylistically consistent artwork by providing reference instances and the original line art. Our code and additional details are available at https://yinhan-zhang.github.io/color.

Figures (6)

• Figure 1: Results of Follow-Your-Color. Given a set of colored references, Follow-Your-Color can colorize a line art image while maintaining color consistency across multiple instances. Compared to traditional methods, our approach significantly improves coloring efficiency.
• Figure 2: Illustration of the workflow of multi-instance sketch colorization production. Previous methods can only achieve multi-instance sketch colorization step by step, which is time-consuming and inaccurate. In contrast, our method can color a sketch while maintaining consistency, making multi-instance sketch colorization easier.
• Figure 3: Overview of the Follow-Your-Color pipeline. We combine a dual-UNet framework with an instance control module (ICM). During training, we use multiple instances to set the overall color accurately. The color matching enables the model to better align the colors of the target image with those of the reference instances precisely. The edge loss helps the model pay more attention to the high-frequency areas and edges, resulting in a more accurate and vivid colorization for each instance.
• Figure 4: Instance Control Ability. With the same line art and diverse reference instances, our method achieves precise instance-level control for varied colorization and all without needing extra guidance.
• Figure 5: Qualitative comparisons with existing methods. Given a line drawing and multiple reference instances, our method demonstrates far more precise colorization and higher-quality results compared to other methods, effectively maintaining line-drawing structure and reference instances' color consistency.