Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StyO: Stylize Your Face in Only One-shot

Bonan Li, Zicheng Zhang, Xuecheng Nie, Congying Han, Yinhan Hu, Xinmin Qiu, Tiande Guo

TL;DR

StyO tackles one-shot face stylization by transferring the target style from a single artistic image while preserving the source face content. It introduces a disentanglement–recombination strategy implemented with Latent Diffusion Models: an Identifier Disentanglement Learner (IDL) produces content and style identifiers via a contrastive prompt and a triple reconstruction loss, and a Fine-grained Content Controller (FCC) recombines them with cross-attention control and augmented prompts to fix fine details. The approach enables exemplar-guided, geometry-aware stylization and surpasses previous baselines in both qualitative and user studies. This work highlights the potential of diffusion-based, text-conditioned stylization with explicit attribute disentanglement for practical, one-shot artistic portrait generation.

Abstract

This paper focuses on face stylization with a single artistic target. Existing works for this task often fail to retain the source content while achieving geometry variation. Here, we present a novel StyO model, ie. Stylize the face in only One-shot, to solve the above problem. In particular, StyO exploits a disentanglement and recombination strategy. It first disentangles the content and style of source and target images into identifiers, which are then recombined in a cross manner to derive the stylized face image. In this way, StyO decomposes complex images into independent and specific attributes, and simplifies one-shot face stylization as the combination of different attributes from input images, thus producing results better matching face geometry of target image and content of source one. StyO is implemented with latent diffusion models (LDM) and composed of two key modules: 1) Identifier Disentanglement Learner (IDL) for disentanglement phase. It represents identifiers as contrastive text prompts, ie. positive and negative descriptions. And it introduces a novel triple reconstruction loss to fine-tune the pre-trained LDM for encoding style and content into corresponding identifiers; 2) Fine-grained Content Controller (FCC) for the recombination phase. It recombines disentangled identifiers from IDL to form an augmented text prompt for generating stylized faces. In addition, FCC also constrains the cross-attention maps of latent and text features to preserve source face details in results. The extensive evaluation shows that StyO produces high-quality images on numerous paintings of various styles and outperforms the current state-of-the-art.

StyO: Stylize Your Face in Only One-shot

TL;DR

StyO tackles one-shot face stylization by transferring the target style from a single artistic image while preserving the source face content. It introduces a disentanglement–recombination strategy implemented with Latent Diffusion Models: an Identifier Disentanglement Learner (IDL) produces content and style identifiers via a contrastive prompt and a triple reconstruction loss, and a Fine-grained Content Controller (FCC) recombines them with cross-attention control and augmented prompts to fix fine details. The approach enables exemplar-guided, geometry-aware stylization and surpasses previous baselines in both qualitative and user studies. This work highlights the potential of diffusion-based, text-conditioned stylization with explicit attribute disentanglement for practical, one-shot artistic portrait generation.

Abstract

This paper focuses on face stylization with a single artistic target. Existing works for this task often fail to retain the source content while achieving geometry variation. Here, we present a novel StyO model, ie. Stylize the face in only One-shot, to solve the above problem. In particular, StyO exploits a disentanglement and recombination strategy. It first disentangles the content and style of source and target images into identifiers, which are then recombined in a cross manner to derive the stylized face image. In this way, StyO decomposes complex images into independent and specific attributes, and simplifies one-shot face stylization as the combination of different attributes from input images, thus producing results better matching face geometry of target image and content of source one. StyO is implemented with latent diffusion models (LDM) and composed of two key modules: 1) Identifier Disentanglement Learner (IDL) for disentanglement phase. It represents identifiers as contrastive text prompts, ie. positive and negative descriptions. And it introduces a novel triple reconstruction loss to fine-tune the pre-trained LDM for encoding style and content into corresponding identifiers; 2) Fine-grained Content Controller (FCC) for the recombination phase. It recombines disentangled identifiers from IDL to form an augmented text prompt for generating stylized faces. In addition, FCC also constrains the cross-attention maps of latent and text features to preserve source face details in results. The extensive evaluation shows that StyO produces high-quality images on numerous paintings of various styles and outperforms the current state-of-the-art.
Paper Structure (11 sections, 7 equations, 7 figures, 1 table)

This paper contains 11 sections, 7 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Framework of StyO
  6. Identifier Disentanglement Learner
  7. Fine-grained Content Controller
  8. Experiments
  9. Comparison with SOTA methods
  10. Ablation Study
  11. Conclusions

Figures (7)

  • Figure 1: Comparison of the proposed StyO and previous state-of-the-art work JoJoGAN Chong2021JoJoGANOS for one-shot face stylization, which aims at applying the style of a single target image to the source one. We can see StyO obviously outperforms JoJoGAN for achieving suitable style from target image, e.g., texture and geometry variations, while maintaining fine-grained content from source image, e.g., head pose and hair style. Better see in color and 2x zoom.
  • Figure 2: Overview of the proposed StyO for one-shot face stylization. StyO consists of two core modules: Identifier Disentanglement Learner (IDL) and Fine-grained Content Controller (FCC). IDL learns to disentangle style and content information of source and target images into different identifiers. Then, FCC recombine disentangled identifiers for fusing source content and target style to generate the stylized facial images.
  • Figure 3: Pipeline of the proposed StyO. Left panel: The training phase. StyO constructs text-image pairs for source, target images and auxiliary image set with a contrastive disentangled prompt template. Then, StyO fine-tunes a pre-trained latent diffusion model in the one-shot manner. In this way, StyO injects attributes of images into different identifiers. Right panel: The inference phase. Given style and content identifiers, StyO recombines them to form text prompt to generating stylized face. To maintain fine-grained details, StyO extracts attention maps for source content and uses them to replace that for stylized one. This process yields faces with suitable target style while maintains good source content.
  • Figure 4: Qualitative comparison of our StyO with GOGA Zhang2022GeneralizedOD, Custom Diffusion kumari2023multi, and StyleID zstar. Better see in color and 2x zoom.
  • Figure 5: Ablation of Contrastive Disentangled Prompt Template. The arrows indicate the changes made to the current prompt template by sequentially plusing positive identifiers, negative identifiers, and auxiliary image set. Better see in color and 2x zoom.
  • ...and 2 more figures