CtlGAN: Few-shot Artistic Portraits Generation with Contrastive Transfer Learning

TL;DR

This work proposes CtlGAN, a new few-shot artistic portraits generation model with a novel contrastive transfer learning strategy that significantly outperforms state-of-the-arts under 10-shot and 1-shot settings and generates high quality artistic portraits.

Abstract

Generating artistic portraits is a challenging problem in computer vision. Existing portrait stylization models that generate good quality results are based on Image-to-Image Translation and require abundant data from both source and target domains. However, without enough data, these methods would result in overfitting. In this work, we propose CtlGAN, a new few-shot artistic portraits generation model with a novel contrastive transfer learning strategy. We adapt a pretrained StyleGAN in the source domain to a target artistic domain with no more than 10 artistic faces. To reduce overfitting to the few training examples, we introduce a novel Cross-Domain Triplet loss which explicitly encourages the target instances generated from different latent codes to be distinguishable. We propose a new encoder which embeds real faces into Z+ space and proposes a dual-path training strategy to better cope with the adapted decoder and eliminate the artifacts. Extensive qualitative, quantitative comparisons and a user study show our method significantly outperforms state-of-the-arts under 10-shot and 1-shot settings and generates high quality artistic portraits. The code will be made publicly available.

Figures (24)

• Figure 1: Our few-shot artistic portraits generation results on different artistic styles ( 10-shot or 1-shot). We eliminate overfitting using a novel contrastive transfer learning strategy. With our style encoder, real face photos are embedded into the latent space shared by our decoders on different artistic domains.
• Figure 2: (a) Few-Shot-GAN-Adaptation ojha2021few-shot-gan results show a certain degree of overfitting (similar faces in the middle row), while ours generates diverse results and well preserves the identity. (b) Results of combining different encoders richardson2021encodinge4eagilegan with our StyleGAN-based artistic decoder. For each input face, the top row shows reconstruction results, and the second row shows cartoonization results.
• Figure 3: Our CtlGAN contains two parts, including a Style Encoder, and a Domain Adaptation Decoder which is based on StyleGAN2 Karras2019stylegan2. We design a dual path training for our encoder with path-1 shown in blue and path-2 (cycle path) shown in orange.
• Figure 4: (a) Few-Shot-GAN-Adaptation ojha2021few-shot-gan framework and (b) Our contrastive transfer learning framework. Given a pretrained source generator and 10 target domain examples, Few-Shot-GAN-Adaptation adapts the model to the target domain by constraining the pairwise similarity before and after adaptation. While we explicitly enforce the target instances generated from two different latent codes to be different to prevent overfitting.
• Figure 5: Comparison with FSGAojha2021few-shot-gan, TGANwang2018transferring, Freeze-Dmo2020freeze, CUTpark2020contrastive under 10 training images setting.