TimeColor: Flexible Reference Colorization via Temporal Concatenation

TL;DR

TimeColor tackles the challenge of sketch-based video colorization with heterogeneous, variable-count references. It introduces a diffusion-transformer framework that encodes references as latent frames concatenated in time and employs modality-disjoint RoPE plus hard spatiotemporal correspondence-masked attention to bind each video region to its designated reference. An automated multi-reference tracking dataset pipeline is built to generate large-scale training data with per-frame correspondence masks. On SAKUGA-42M, TimeColor delivers superior color fidelity, identity preservation, and temporal stability, especially in multi-reference settings, highlighting its potential for production workflows with diverse reference sources.

Abstract

Most colorization models condition only on a single reference, typically the first frame of the scene. However, this approach ignores other sources of conditional data, such as character sheets, background images, or arbitrary colorized frames. We propose TimeColor, a sketch-based video colorization model that supports heterogeneous, variable-count references with the use of explicit per-reference region assignment. TimeColor encodes references as additional latent frames which are concatenated temporally, permitting them to be processed concurrently in each diffusion step while keeping the model's parameter count fixed. TimeColor also uses spatiotemporal correspondence-masked attention to enforce subject-reference binding in addition to modality-disjoint RoPE indexing. These mechanisms mitigate shortcutting and cross-identity palette leakage. Experiments on SAKUGA-42M under both single- and multi-reference protocols show that TimeColor improves color fidelity, identity consistency, and temporal stability over prior baselines.

Figures (14)

• Figure 1: TimeColor enables sketch video colorization with a fixed parameter budget, conditioning on heterogeneous, variable-count references. It generates identity-consistent, temporally stable colorized animations from sketch videos, aiming to reduce manual 2D colorization effort.
• Figure 2: Overview of TimeColor. Given a sketched video and a variable-length reference bank (starting-frame, arbitrary-frame, and multi-reference cues), TimeColor conditions a DiT video diffusion model via temporal token concatenation, modality-disjoint RoPE, and correspondence-masked attention to bind subjects to references while mitigating shortcutting/identity leakage, enabling flexible reference types and lengths with a fixed backbone and parameter count.
• Figure 3: TimeColor model diagram. Target video, sketch, and a variable set of image reference tokens are temporally concatenated. Correspondence-masked attention restricts attention to the assigned reference, enforcing strict reference correspondence. Unlike adapter/channel-stacking controls, TimeColor supports variable reference counts with fixed parameters and concurrent reference conditioning, and remains robust to non-starting-frame misalignment.
• Figure 4: Qualitative comparison among baselines under single-reference (starting-frame and arbitrary-frame) and multi-reference settings across seven methods: VACE Jiang2025VACE, LVCD Huang2024LVCD, AniDoc Meng2025AniDoc, ToonCrafter Xing2024ToonCrafter, ToonComposer li2025tooncomposer, LongAnimation Chen2025LongAnimation, and TimeColor.
• Figure 5: Robustness to mismatched viewpoints. With large pose/viewpoint gaps between references and targets, TimeColor maintains temporal coherence and palette fidelity while avoiding cross-reference leakage.