Sketch2Manga: Shaded Manga Screening from Sketch with Diffusion Models

TL;DR

A novel sketch-to-manga framework that first generates a color illustration from the sketch and then generates a screentoned manga based on the intensity guidance significantly outperforms existing methods in generating high-quality manga with shaded high-frequency screentones.

Abstract

While manga is a popular entertainment form, creating manga is tedious, especially adding screentones to the created sketch, namely manga screening. Unfortunately, there is no existing method that tailors for automatic manga screening, probably due to the difficulty of generating high-quality shaded high-frequency screentones. The classic manga screening approaches generally require user input to provide screentone exemplars or a reference manga image. The recent deep learning models enables the automatic generation by learning from a large-scale dataset. However, the state-of-the-art models still fail to generate high-quality shaded screentones due to the lack of a tailored model and high-quality manga training data. In this paper, we propose a novel sketch-to-manga framework that first generates a color illustration from the sketch and then generates a screentoned manga based on the intensity guidance. Our method significantly outperforms existing methods in generating high-quality manga with shaded high-frequency screentones.

Figures (6)

• Figure 1: System overview. Given an input sketch, our system first generates a color illustration from the sketch and then generates a rough manga image using a diffusion model conditioned on the intensity map of the color illustration. An adaptive scaling process is further applied to generate the final manga image.
• Figure 2: Ablation on latent diffusion. The original diffusion model fails to generate high-frequency screentones without finetuning. After finetuning, conditioning on sketch still leads to flat shading, while conditioning on intensity generates shaded high-frequency screentones.
• Figure 3: Ablation on VAE decoder. The original VAE decoder fails to generate visually pleasant screetones. Finetuning the VAE decoder without LPIP loss produces the best results.
• Figure 4: Example of adaptive scaling on lightness (L) and saturation (S) respectively. Scaling on saturation achieves better results by preserving the shading of the color illustration.
• Figure 5: Comparisons with state-of-the-art sketch-to-manga methods.