Stylized Face Sketch Extraction via Generative Prior with Limited Data

TL;DR

It is shown that StyleSketch outperforms existing state‐of‐the‐art sketch extraction methods and few‐shot image adaptation methods for the task of extracting high‐resolution abstract face sketches and is extended by extending its use to other domains and exploring the possibility of semantic editing.

Abstract

Facial sketches are both a concise way of showing the identity of a person and a means to express artistic intention. While a few techniques have recently emerged that allow sketches to be extracted in different styles, they typically rely on a large amount of data that is difficult to obtain. Here, we propose StyleSketch, a method for extracting high-resolution stylized sketches from a face image. Using the rich semantics of the deep features from a pretrained StyleGAN, we are able to train a sketch generator with 16 pairs of face and the corresponding sketch images. The sketch generator utilizes part-based losses with two-stage learning for fast convergence during training for high-quality sketch extraction. Through a set of comparisons, we show that StyleSketch outperforms existing state-of-the-art sketch extraction methods and few-shot image adaptation methods for the task of extracting high-resolution abstract face sketches. We further demonstrate the versatility of StyleSketch by extending its use to other domains and explore the possibility of semantic editing. The project page can be found in https://kwanyun.github.io/stylesketch_project.

Figures (12)

• Figure 1: Data-Artifact Tradeoff: Domain adaptation methods may require less data but often exhibit color bleeding chong2022jojogan. Image translation-based sketch extraction methods avoid such artifacts, but require a large dataset ashtari2022reference. Our approach, using deep features from generative models while training from scratch, avoids artifacts and can be trained with a limited number of data.
• Figure 2: Overview of StyleSketch. To extract a sketch from a face image, we first train a sketch generator $G_{sketch}$ that accepts deep features as input. $G_{sketch}$ is constructed with the deep fusion module represented on the right side of the figure. After training, we extract sketches by performing GAN inversion of the input image followed by feeding the deep features into $G_{sketch}$.
• Figure 3: Results produced according to the iterations performed with and without the initial stage. Training without L1 loss requires a large number of iterations to optimize the output into the sketch domain.
• Figure 4: Overview of our authentic face sketch dataset (SKSF-A). The dataset consists of photo and sketch pairs while the sketch has one of 7 different styles.
• Figure 5: Comparison with domain adaptation baseline methods. All methods are trained with 16 data. Our method produced the best quality results among all the methods while other methods suffer from identity shift or color bleeding.