Training-free Stylized Text-to-Image Generation with Fast Inference

TL;DR

This work tackles stylized text-to-image generation with large diffusion models under practical constraints by avoiding expensive inversion or fine-tuning. It introduces OmniPainter, which uses Latent Consistency Models to extract representative style statistics from a single reference image and then employs a norm-based, AdaIN-informed self-attention mechanism to fuse style with content during a short denoising process. The key contributions are a training-free, inversion-free stylization pipeline, the extraction of style statistics without DDIM inversion, and the Norm Mixture of Self-Attention that preserves content fidelity while achieving strong style alignment; experiments show superior style consistency and competitive content fidelity with an average inference time of about 0.7 seconds. The approach significantly improves the practicality of stylized diffusion-based generation, enabling rapid, flexible style control for broad, real-time applications.

Abstract

Although diffusion models exhibit impressive generative capabilities, existing methods for stylized image generation based on these models often require textual inversion or fine-tuning with style images, which is time-consuming and limits the practical applicability of large-scale diffusion models. To address these challenges, we propose a novel stylized image generation method leveraging a pre-trained large-scale diffusion model without requiring fine-tuning or any additional optimization, termed as OmniPainter. Specifically, we exploit the self-consistency property of latent consistency models to extract the representative style statistics from reference style images to guide the stylization process. Additionally, we then introduce the norm mixture of self-attention, which enables the model to query the most relevant style patterns from these statistics for the intermediate output content features. This mechanism also ensures that the stylized results align closely with the distribution of the reference style images. Our qualitative and quantitative experimental results demonstrate that the proposed method outperforms state-of-the-art approaches.

Figures (15)

• Figure 1: Examples generated by our method using different paintings of Van Gogh as style images.
• Figure 2: Examples of using the style transfer method for stylized T2I generation directly. We first generate images from prompts using the T2I method luo2023latent, then apply style transfer methods deng2023zliu2024zepo to incorporate the specified style.
• Figure 3: The overall pipeline of our method. Here, $\sigma$, "Repre style statistics", and "Cont features" are the softmax operation, representative style statistics, and content features, respectively. The whole stylization process operates in the latent space of the pre-trained VAE.
• Figure 4: CLIP features similarity of different combinations at different timesteps.
• Figure 5: Issues of the direct replacing method and visualization of the top three leading components.