InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser

TL;DR

This work introduces InstaStyle, a diffusion-based method that leverages the inversion noise of a single stylized reference image to generate high-fidelity stylized images. By applying DDIM inversion, the authors reveal a non-zero signal-to-noise ratio in the inversion noise, indicating retention of the reference style; they further refine a learnable style token via prompt refinement and human feedback to overcome ambiguities in natural language prompts. The approach uses a two-stage process: initial stylized image generation from inversion noise and subsequent style-token learning with LoRA-based cross-attention tuning, enabling precise and flexible style control, including style combination. Empirical results demonstrate superior style fidelity and content accuracy against baselines, with qualitative and quantitative evidence and user studies supporting the method’s effectiveness and its creative potential for combining styles.

Abstract

Stylized text-to-image generation focuses on creating images from textual descriptions while adhering to a style specified by a few reference images. However, subtle style variations within different reference images can hinder the model from accurately learning the target style. In this paper, we propose InstaStyle, a novel approach that excels in generating high-fidelity stylized images with only a single reference image. Our approach is based on the finding that the inversion noise from a stylized reference image inherently carries the style signal, as evidenced by their non-zero signal-to-noise ratio. We employ DDIM inversion to extract this noise from the reference image and leverage a diffusion model to generate new stylized images from the "style" noise. Additionally, the inherent ambiguity and bias of textual prompts impede the precise conveying of style. To address this, we introduce a learnable style token via prompt refinement, which enhances the accuracy of the style description for the reference image. Qualitative and quantitative experimental results demonstrate that InstaStyle achieves superior performance compared to current benchmarks. Furthermore, our approach also showcases its capability in the creative task of style combination with mixed inversion noise.

Figures (21)

• Figure 1: Visualization of InstaStyle. (a) Our method excels at capturing style details and distinguishing between similar styles. (b) The first and third columns show images styled with reference to style 1 and style 2, respectively. The middle column shows images in a combined style. (c) Our method supports adjusting the degree of two styles during combination, dynamically ranging from one style to another.
• Figure 2: Motivation. Sampling from the inversion noise of a reference image can generate stylized images. However, the optimal style token varies for each case among human-written style tokens. Our learnable style token, i.e., "< style1>", shows greater universality across various scenarios.
• Figure 3: The training process of InstaStyle. (a) The first stage is initial stylized image generation. The reference image is inverted to noise conditioned on a prompt via DDIM Inversion. Then the inversion noise is utilized to generate initial stylized images. (b) The second stage is prompt refinement which leverages the selected high-quality initial stylized images to learn a style token.
• Figure 4: Qualitative comparison of stylized image generation on various styles. Objects for synthesis are Bicycle, Sunflowers and Chair. Our method excels at capturing fine-grained style information, such as color, textures, and brushstrokes.
• Figure 5: Qualitative comparison with style transfer methods. For style transfer methods, content images are employed (shown in the bottom right). Despite our method relying solely on text for content, we achieve comparable performance in the fidelity of the content. Furthermore, we excel in preserving style details.