StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance

TL;DR

This work tackles content leakage in visual style prompting for text-to-image diffusion models by introducing StyleKeeper, a training-free framework that combines classifier-free guidance with swapped self-attention, negative visual query guidance, stochastic encoding of real images, and color calibration. The method selectively swaps high-level upblock self-attention, uses a KV-injected score to separate style from content, and employs NVQG to suppress unwanted visual content, achieving faithful text-driven content while reflecting reference style. Real-image prompts are supported via stochastic encoding and color calibration to maintain style fidelity without costly inversions. Empirical results across diverse prompts and baselines show improved style similarity, preserved content fidelity, and reduced content leakage, with demonstrated compatibility with ControlNet and other I2I editing approaches, highlighting practical implications for flexible, training-free visual style control.

Abstract

In the domain of text-to-image generation, diffusion models have emerged as powerful tools. Recently, studies on visual prompting, where images are used as prompts, have enabled more precise control over style and content. However, existing methods often suffer from content leakage, where undesired elements of the visual style prompt are transferred along with the intended style. To address this issue, we 1) extend classifier-free guidance (CFG) to utilize swapping self-attention and propose 2) negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. NVQG employs negative score by intentionally simulating content leakage scenarios that swap queries instead of key and values of self-attention layers from visual style prompts. This simple yet effective method significantly reduces content leakage. Furthermore, we provide careful solutions for using a real image as visual style prompts. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, reflecting the style of the references, and ensuring that resulting images match the text prompts. Our code is available \href{https://github.com/naver-ai/StyleKeeper}{here}.

Figures (36)

• Figure 1: Qualitative comparison with the same style. Competitors face challenges in generating images with diverse layouts and compositions, i.e., content leakage from the reference.
• Figure 2: Overveiw of StyleKeeper. Our proposed method includes 4 key components, highlighted in red boxes. First, stochastic encoding (Section \ref{['sec:real_image_as_reference']}) converts reference images into suitable latents for the visual style prompting task. Second, swapping self-attention (Section \ref{['sec:swapping_selfattention']}, \ref{['sec:choosing_blocks']}) ensures the reference image’s style is accurately reflected. Third, negative visual query guidance (Section \ref{['sec:negativequeryguidance']}) reduces content leakage from the reference image, allowing the desired text content (e.g., "Moose") to be better represented. Lastly, color calibration (Section \ref{['sec:real_image_as_reference']}) minimizes errors during the denoising process, helping to produce a cleaner final image.
• Figure 3: The effect of swapping self-attention across different blocks. Swapping self-attention on the bottleneck and downblocks causes content leakage, resulting in cat-like images despite a dog prompt, while swapping on downblocks disrupts resulting images. We only apply swapping self-attention in the upblocks to reflect style elements effectively.
• Figure 4: The effect of CFG and the proposed negative visual query guidance on image generation. The reference images provide the style for each generated output. Without NVQG, content leakage occurs, and the generated images fail to fully capture the intended content. In contrast, using NVQG ensures better alignment with both the reference style and the "Cat" prompt, reducing content distortion and improving quality.
• Figure 5: Analysis on the optimal range of upblocks for swapping self-attention. We find the optimal range of upblocks for a balanced trade-off between different aspects. The images on the right illustrate the visual results for different upblock layer indices, with the red cross indicating poor diversity and misalignment to the text prompt, the red triangle indicating a lack of style similarity, and the yellow star indicating the optimal results. Please refer to Section \ref{['sec:exp_swap_sa']} for details.