Computer-aided Colorization State-of-the-science: A Survey

TL;DR

This survey maps computer-aided colorization into conditional, unconditional, and video categories, detailing how control signals—from references and hints to text and multi-modal cues—have evolved alongside advances in deep learning, segmentation, and diffusion models. It extends traditional evaluation by introducing colorization-specific aesthetic assessment based on CLIP-IQA and LAION-Aesthetics, revealing that perceptual quality and colorfulness do not always align with reconstruction metrics. Datasets and applications across natural images, line drawings, and video are discussed, with emphasis on cross-domain challenges and the need for robust generalization. The discussion points toward integrating colorization with AIGC workflows, multi-modal interaction, and diffusion-based editing as promising future directions, culminating in a taxonomy and state-of-the-art benchmarks like DDColor.

Abstract

This paper reviews published research in the field of computer-aided colorization technology. We argue that the colorization task originates from computer graphics, prospers by introducing computer vision, and tends to the fusion of vision and graphics, so we put forward our taxonomy and organize the whole paper chronologically. We extend the existing reconstruction-based colorization evaluation techniques, considering that aesthetic assessment of colored images should be introduced to ensure that colorization satisfies human visual-related requirements and emotions more closely. We perform the colorization aesthetic assessment on seven representative unconditional colorization models and discuss the difference between our assessment and the existing reconstruction-based metrics. Finally, this paper identifies unresolved issues and proposes fruitful areas for future research and development. Access to the project associated with this survey can be obtained at https://github.com/DanielCho-HK/Colorization.

Figures (10)

• Figure 1: Taxonomy of colorization technology. We classify colorization as a sub-task of computer graphics, in which the green part belongs to the explicit diverse colorization using conditional controls, and the blue part belongs to the colorization methods and applications using computer vision technology. We individually mark the video colorization in yellow to represent it as an extension of image colorization in spatiotemporal dimensions.
• Figure 2: Illustration of reference-based colorization. The top row shows reference-based gray-scale image colorization and the bottom row shows reference-based line drawing colorization. Images courtesy of Welsh02cao23.
• Figure 3: Illustration of colorization using color scribbles (or strokes). The top row shows gray-scale image colorization, and the bottom row shows the colorization of line drawings. Images courtesy of levin04qu06.
• Figure 4: Illustration of colorization using color points. The top row shows gray-scale image colorization, and the bottom row shows the colorization of the line drawing. Images courtesy of zhang17zhang18.
• Figure 5: Palette-based colorization method is suitable for image recolorization applications. Users can directly change one or all color elements of the palette extracted from the input image. The black line below the color element indicates the color users select to edit. Images courtesy of chang2015palette.