Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ZePo: Zero-Shot Portrait Stylization with Faster Sampling

Jin Liu, Huaibo Huang, Jie Cao, Ran He

TL;DR

ZePo tackles the bottleneck of slow, inversion-dependent portrait stylization in diffusion models by eliminating the need for fine-tuning and DDIM inversion. It leverages Latent Consistency Models to extract Consistency Features from content and reference images and introduces Style Enhancement Attention Control to fuse these features efficiently during four-sample generation. The approach yields fast, high-fidelity stylization with strong content preservation and style transfer, validated by quantitative metrics (LPIPS, CLIP-IQA) and qualitative comparisons, while maintaining practical inference times (~0.6 seconds). The authors provide ablations and comparisons against state-of-the-art baselines and release the code for public use.

Abstract

Diffusion-based text-to-image generation models have significantly advanced the field of art content synthesis. However, current portrait stylization methods generally require either model fine-tuning based on examples or the employment of DDIM Inversion to revert images to noise space, both of which substantially decelerate the image generation process. To overcome these limitations, this paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We observed that Latent Consistency Models employing consistency distillation can effectively extract representative Consistency Features from noisy images. To blend the Consistency Features extracted from both content and style images, we introduce a Style Enhancement Attention Control technique that meticulously merges content and style features within the attention space of the target image. Moreover, we propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control. Extensive experiments have validated the effectiveness of our proposed framework in enhancing stylization efficiency and fidelity. The code is available at \url{https://github.com/liujin112/ZePo}.

ZePo: Zero-Shot Portrait Stylization with Faster Sampling

TL;DR

ZePo tackles the bottleneck of slow, inversion-dependent portrait stylization in diffusion models by eliminating the need for fine-tuning and DDIM inversion. It leverages Latent Consistency Models to extract Consistency Features from content and reference images and introduces Style Enhancement Attention Control to fuse these features efficiently during four-sample generation. The approach yields fast, high-fidelity stylization with strong content preservation and style transfer, validated by quantitative metrics (LPIPS, CLIP-IQA) and qualitative comparisons, while maintaining practical inference times (~0.6 seconds). The authors provide ablations and comparisons against state-of-the-art baselines and release the code for public use.

Abstract

Diffusion-based text-to-image generation models have significantly advanced the field of art content synthesis. However, current portrait stylization methods generally require either model fine-tuning based on examples or the employment of DDIM Inversion to revert images to noise space, both of which substantially decelerate the image generation process. To overcome these limitations, this paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We observed that Latent Consistency Models employing consistency distillation can effectively extract representative Consistency Features from noisy images. To blend the Consistency Features extracted from both content and style images, we introduce a Style Enhancement Attention Control technique that meticulously merges content and style features within the attention space of the target image. Moreover, we propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control. Extensive experiments have validated the effectiveness of our proposed framework in enhancing stylization efficiency and fidelity. The code is available at \url{https://github.com/liujin112/ZePo}.
Paper Structure (15 sections, 10 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 10 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Few-Shot Face Stylization
  4. Diffusion-Based Style Transfer
  5. Preliminaries
  6. Latent Diffusion Models
  7. Latent Consistency Models
  8. Method
  9. Consistency Features
  10. Style Enhancement Attention Control
  11. Experiments
  12. Qualitative Comparison
  13. Quantitative Comparison.
  14. Ablation Study
  15. Conclusion

Figures (7)

  • Figure 1: The overall framework of ZePo. The framework is divided into two stages. The first stage involves the extraction of consistency features, where multi-scale consistent features are extracted from the reference and source images with slight noise added. The second stage is the stylized image synthesis phase, where the source image, added with a moderate level of noise, is used as the input. In this phase, the Style Enhancement Attention Control module within the U-Net fuses the consistency features from both the reference and source images to synthesize a stylized portrait.
  • Figure 2: The results of one-step denoising with different noise levels (time-step), different noise addition methods (DDIM Inversion and Forward Process), and different models (SD and LCM) are examined. (a) DDIM Inversion + SD. (b) Forward Process + SD. (c) Forward Process + LCM.
  • Figure 3: Qualitative comparisons with conventional portrait stylization baselines. (a) and (b) are the input reference image and content image, respectively, while (c-h) are the stylized portraits synthesized by different baselines.
  • Figure 4: Ablation experiments on Attention Control.
  • Figure 5: Ablation experiment on sampling time steps. Our method produces satisfactory stylized results with just one sampling step. Increasing the number of sampling steps further enhances the detail in the synthesized results.
  • ...and 2 more figures