Learning Inclusion Matching for Animation Paint Bucket Colorization

TL;DR

This work addresses labor-intensive paint bucket colorization of animation line art by introducing an inclusion matching paradigm that learns region-level affiliations rather than strict segment-to-segment correspondences. A coarse-to-fine two-stage pipeline combines RAFT-based color warping with a deformable-convolution–CLIP–multiplex-transformer framework to propagate colors across frames, guided by an inclusion-based loss and a newly released PaintBucket-Character dataset. The approach demonstrates robust performance under occlusion and large motion, outperforming segment-matching and optical-flow baselines and offering practical improvements for industrial animation workflows. The dataset and methodology collectively advance automated, reliable line-art colorization with potential for integration into production pipelines, reducing manual workload and color inconsistencies in hand-drawn animation.

Abstract

Colorizing line art is a pivotal task in the production of hand-drawn cel animation. This typically involves digital painters using a paint bucket tool to manually color each segment enclosed by lines, based on RGB values predetermined by a color designer. This frame-by-frame process is both arduous and time-intensive. Current automated methods mainly focus on segment matching. This technique migrates colors from a reference to the target frame by aligning features within line-enclosed segments across frames. However, issues like occlusion and wrinkles in animations often disrupt these direct correspondences, leading to mismatches. In this work, we introduce a new learning-based inclusion matching pipeline, which directs the network to comprehend the inclusion relationships between segments rather than relying solely on direct visual correspondences. Our method features a two-stage pipeline that integrates a coarse color warping module with an inclusion matching module, enabling more nuanced and accurate colorization. To facilitate the training of our network, we also develope a unique dataset, referred to as PaintBucket-Character. This dataset includes rendered line arts alongside their colorized counterparts, featuring various 3D characters. Extensive experiments demonstrate the effectiveness and superiority of our method over existing techniques.

Figures (17)

• Figure 1: In the animation industry, digital painters use paint bucket tool to colorize drawn line arts frame by frame. Our proposed pipeline streamlines this process by requiring the painters to colorize just one frame, after which the algorithm autonomously propagates the color to subsequent frames, enabling automatic colorization. Compared with optical-flow-based method RAFT teed2020raft and segment-matching-based method AnT AnT, our method can achieve more robust results on challenging cases such as one-to-many matching, large deformation, and tiny region colorization. In this figure, RAFT is trained on Sintel dataset sintel and finetuned on AnimeRun siyao2022animerun. We use the most frequent color in each segment to colorize each line-enclosed region. © drawn by Nicca (Sriprachum Kongwisawamit), used with artist permission.
• Figure 2: Methods relying on segment matching typically seek the most similar segment across frames, yet challenges arise in scenarios involving occlusion and wrinkles. This is particularly evident in tiny segments, as highlighted in the red box, where disruptions in correspondence lead to mismatches. Our innovative approach, based on inclusion matching, addresses this issue by estimating the inclusion relationship rather than pursuing direct correspondence, as illustrated by the comparison of red and blue matching lines.
• Figure 3: Several examples of our rendered characters in PaintBucket-Character dataset.
• Figure 4: Overview of our synthetic data generation pipeline. We extract the UV map (a) and the used UV region from the character. Then, we use a paint bucket tool to fill the used UV region, creating color label (b) and index label (c). Label images (b) and (c) are then pasted back to the 3D meshes to create flat color characters (d) and (e), respectively. Finally, we post-process the index image (e) to obtain the rendered line art image (f).
• Figure 5: Our simulation method can generate color lines that closely mimic hand-drawn line arts. Red, blue, and green lines represent highlights, shadows, and other instructions respectively.