PS-StyleGAN: Illustrative Portrait Sketching using Attention-Based Style Adaptation

TL;DR

This paper introduces PS-StyleGAN, a style transfer approach tailored for portrait sketch synthesis that uses only a few paired examples to model a style and has a short training time, and demonstrates its superiority over the current state-of-the-art methods on various datasets, qualitatively and quantitatively.

Abstract

Portrait sketching involves capturing identity specific attributes of a real face with abstract lines and shades. Unlike photo-realistic images, a good portrait sketch generation method needs selective attention to detail, making the problem challenging. This paper introduces \textbf{Portrait Sketching StyleGAN (PS-StyleGAN)}, a style transfer approach tailored for portrait sketch synthesis. We leverage the semantic $W+$ latent space of StyleGAN to generate portrait sketches, allowing us to make meaningful edits, like pose and expression alterations, without compromising identity. To achieve this, we propose the use of Attentive Affine transform blocks in our architecture, and a training strategy that allows us to change StyleGAN's output without finetuning it. These blocks learn to modify style latent code by paying attention to both content and style latent features, allowing us to adapt the outputs of StyleGAN in an inversion-consistent manner. Our approach uses only a few paired examples ($\sim 100$) to model a style and has a short training time. We demonstrate PS-StyleGAN's superiority over the current state-of-the-art methods on various datasets, qualitatively and quantitatively.

Figures (6)

• Figure 0: Outputs of PS-StyleGAN for different sketching styles (inset) in specified poses and expressions while maintaining the input identity. A model trained on FS2K dataset was used for (b) - (d), while CUHK and APDrawing were used for the models in (e) and (f).
• Figure 1: Results of DualStyleGAN trained on CUHK cuhk dataset. The generated sketch (b) is a result of complete structure and color transfer. Structure transfer (c) results in considerable loss of identity while color transfer (d) does not yield stylization.
• Figure 2: An overview of our model architecture. We use a pretrained 256x256 resolution StyleGAN2 Karras_2019_CVPR generator $g$ fitted with three style adaptation blocks at the fine resolution layers. Each block consists of a novel Attentive Affine transform module ($\mathbb{A}$) that predicts affine parameters from attention-weighted latent codes of $S$ using supervision from $w_c^+$ and $w_s^+$. These parameters are then used to modulate and normalize the spatial features of $g$ at different scales to imbibe the style $\mathbb{S}$ into $C$.
• Figure 3: (a) The structure of AdaIN AdaIN module used in StyleGAN Karras2018ASG. (b) The structure of AdaAttN AdaAttN module. (c) The structure of our proposed design showing attentive affine transform blocks. Here, A denotes a basic affine transform block consisting of a single trainable fully-connected layer and Norm denotes channel-wise mean-variance normalization.
• Figure 4: Results after each stage of progressive transfer learning. At the end of stage I, the model converges to an average representative style as seen in (b) where the eyes, nose and mouth are sketched in a similar manner. Stage II widens the model's generative space to capture subtle style variations resulting in better identity preservation as shown in (c).