MagicStyle: Portrait Stylization Based on Reference Image

TL;DR

This work proposes a diffusion model-based reference image stylization method specifically for portraits, called MagicStyle, and conducts comprehensive comparative and ablation experiments to validate the effectiveness of the proposed MagicStyle and FFA.

Abstract

The development of diffusion models has significantly advanced the research on image stylization, particularly in the area of stylizing a content image based on a given style image, which has attracted many scholars. The main challenge in this reference image stylization task lies in how to maintain the details of the content image while incorporating the color and texture features of the style image. This challenge becomes even more pronounced when the content image is a portrait which has complex textural details. To address this challenge, we propose a diffusion model-based reference image stylization method specifically for portraits, called MagicStyle. MagicStyle consists of two phases: Content and Style DDIM Inversion (CSDI) and Feature Fusion Forward (FFF). The CSDI phase involves a reverse denoising process, where DDIM Inversion is performed separately on the content image and the style image, storing the self-attention query, key and value features of both images during the inversion process. The FFF phase executes forward denoising, harmoniously integrating the texture and color information from the pre-stored feature queries, keys and values into the diffusion generation process based on our Well-designed Feature Fusion Attention (FFA). We conducted comprehensive comparative and ablation experiments to validate the effectiveness of our proposed MagicStyle and FFA.

Figures (5)

• Figure 1: Flowchart of MagicStyle. Left side illustrates the Content and Style DDIM Inversion (CSDI) process, and the self-attention features will be restored for the right side Feature Fusion Forward (FFF) process. The fusion operation is mainly conducted during Feature Fusion Attention (FFA).
• Figure 2: Generation results of MagicStyle with various of contents and styles, $\alpha=0.8, \beta=0.2$. MagicStyle can generate stylized images for portrait content images of different genders, ages and colors, as well as for various style images.
• Figure 3: Generation results of MagicStyle with the face resolution in content image is relatively small, $\alpha=0.8, \beta=0.2$. The second row of each content image are results in coorperated with DIIR proposed in MagicID deng2024magicid.
• Figure 4: Visualization comparison with baseline models, MagicStyle ($\alpha=0.8, \beta=0.2$) can not only provide excellent image stylization results, but also preserve details such as facial identity, expressions, and background from the content image.
• Figure 5: Ablation results of Feature Fusion Attention(FFA). We adjust the weights of the content and style queries ($\alpha$ and $\beta$), while maintaining $\alpha+\beta=1$. By increasing $\beta$, we explore the impact of these two weights on image stylization in MagicStyle. Additionally, we replace FFA with standard Attention to validate the stylization effectiveness of FFA.