Z-STAR+: A Zero-shot Style Transfer Method via Adjusting Style Distribution

TL;DR

Z-STAR+ addresses zero-shot style transfer by exploiting latent style and content distributions within pre-trained diffusion models, avoiding retraining. It introduces a dual-path inversion pipeline to encode content and style, a Cross-attention Reweighting module for precise local style alignment, and a Scaled Adaptive Instance Normalization to harmonize global color distribution, with an extension to video style transfer through inter-frame coherence. The approach demonstrates improved style fidelity and content preservation, supported by qualitative and quantitative evaluations and comprehensive ablations that validate the necessity of its components. The work offers a practical, training-free solution for high-quality image and video style transfer with flexible regional and conditional control.

Abstract

Style transfer presents a significant challenge, primarily centered on identifying an appropriate style representation. Conventional methods employ style loss, derived from second-order statistics or contrastive learning, to constrain style representation in the stylized result. However, these pre-defined style representations often limit stylistic expression, leading to artifacts. In contrast to existing approaches, we have discovered that latent features in vanilla diffusion models inherently contain natural style and content distributions. This allows for direct extraction of style information and seamless integration of generative priors into the content image without necessitating retraining. Our method adopts dual denoising paths to represent content and style references in latent space, subsequently guiding the content image denoising process with style latent codes. We introduce a Cross-attention Reweighting module that utilizes local content features to query style image information best suited to the input patch, thereby aligning the style distribution of the stylized results with that of the style image. Furthermore, we design a scaled adaptive instance normalization to mitigate inconsistencies in color distribution between style and stylized images on a global scale. Through theoretical analysis and extensive experimentation, we demonstrate the effectiveness and superiority of our diffusion-based \uline{z}ero-shot \uline{s}tyle \uline{t}ransfer via \uline{a}djusting style dist\uline{r}ibution, termed Z-STAR+.

Figures (18)

• Figure 1: Different pipline for style transfer task. The schematic stylized results are generated by StyTr$^2$Deng:2022:CVPR, InST zhang:2023:inversion, InstantStyle-Plus instancestyle and our method.
• Figure 2: Overall pipeline of our style transfer framework. The stylization process operates in the latent space. The process begins with separate DDIM inversions for both the content and style images. Firstly, we use the SAIN to adjust global style distribution in the initial step. During the subsequent denoising process, we employ our novel Cross-attention Reweighting technique to integrate local style patterns into the content structure. The stylization is achieved through an iterative process of 30 denoising steps, ultimately producing the final stylized output.
• Figure 3: The results obtained using style-cross attention demonstrate a tendency to overemphasize stylistic elements at the expense of preserving original content structures.Distortions are observed in areas where there is low correlation between self-attention and cross-attention outputs, indicating that $V_s$ inadequately reconstructs these target regions. "Simple Addition" retains an excessive amount of content features, whereas the Cross-attention Reweighting achieves a optimal trade-off.
• Figure 5: The t-SNE visualization using mean and various latent features of a set of style images.
• Figure 6: Style difference between stylized/content features and style features in the denoising process. The images enclosed in yellow boxes represent visualized style features, while those in green boxes depict visualized content features. Images within red boxes showcase stylized features obtained through local style adjustment. The images contained in blue boxes demonstrate stylized features resulting from both local and global adjustments..