Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration

Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin

TL;DR

Sissi tackles the challenge of precise style-guided image synthesis without model retraining by recasting style integration as an in-context learning task using a ReFlow-based inpainting backbone. It introduces Dynamic Semantic-Style Integration (DSSI), which adaptively balances text-driven semantic guidance and style cues from a reference image within a Diffusion Transformer–based architecture, mitigating cross-modal conflicts and noise sensitivity. The method demonstrates strong qualitative and quantitative stylization with content fidelity, generalizes to multiple styles, and supports image editing tasks, all in a training-free setting. This approach offers a practical, efficient alternative to retraining or inversion-based methods, with broad applicability to diverse stylization and editing scenarios.

Abstract

Text-guided image generation has advanced rapidly with large-scale diffusion models, yet achieving precise stylization with visual exemplars remains difficult. Existing approaches often depend on task-specific retraining or expensive inversion procedures, which can compromise content integrity, reduce style fidelity, and lead to an unsatisfactory trade-off between semantic prompt adherence and style alignment. In this work, we introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task. Guided by textual semantic prompts, our method concatenates a reference style image with a masked target image, leveraging a pretrained ReFlow-based inpainting model to seamlessly integrate semantic content with the desired style through multimodal attention fusion. We further analyze the imbalance and noise sensitivity inherent in multimodal attention fusion and propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between textual semantic and style visual tokens, effectively resolving guidance conflicts and enhancing output coherence. Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality, offering a simple yet powerful alternative to complex, artifact-prone prior methods.

Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration

TL;DR

Sissi tackles the challenge of precise style-guided image synthesis without model retraining by recasting style integration as an in-context learning task using a ReFlow-based inpainting backbone. It introduces Dynamic Semantic-Style Integration (DSSI), which adaptively balances text-driven semantic guidance and style cues from a reference image within a Diffusion Transformer–based architecture, mitigating cross-modal conflicts and noise sensitivity. The method demonstrates strong qualitative and quantitative stylization with content fidelity, generalizes to multiple styles, and supports image editing tasks, all in a training-free setting. This approach offers a practical, efficient alternative to retraining or inversion-based methods, with broad applicability to diverse stylization and editing scenarios.

Abstract

Text-guided image generation has advanced rapidly with large-scale diffusion models, yet achieving precise stylization with visual exemplars remains difficult. Existing approaches often depend on task-specific retraining or expensive inversion procedures, which can compromise content integrity, reduce style fidelity, and lead to an unsatisfactory trade-off between semantic prompt adherence and style alignment. In this work, we introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task. Guided by textual semantic prompts, our method concatenates a reference style image with a masked target image, leveraging a pretrained ReFlow-based inpainting model to seamlessly integrate semantic content with the desired style through multimodal attention fusion. We further analyze the imbalance and noise sensitivity inherent in multimodal attention fusion and propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between textual semantic and style visual tokens, effectively resolving guidance conflicts and enhancing output coherence. Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality, offering a simple yet powerful alternative to complex, artifact-prone prior methods.
Paper Structure (26 sections, 3 theorems, 23 equations, 13 figures, 3 tables)

This paper contains 26 sections, 3 theorems, 23 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalized Image Generation
  4. Style Transfer
  5. Preliminaries
  6. Rectified Flow Models
  7. Diffusion Transformer Structure
  8. Method
  9. In-context Style Enhancement
  10. Problems in the Vanilla Setting
  11. Inherent Imbalance in Semantic-Style Contributions
  12. Sensitivity to Noise
  13. Dynamic Semantic-Style Integration
  14. Noise Sensitivity Analysis
  15. Experiment
  16. ...and 11 more sections

Key Result

Proposition 1

Assume that for each branch $b \in \{p,s,o\}$, the entries of $\mathbf{Z}$ satisfy $\mathbf{Z}_{i,j} = \mu_b + \xi_{i,j}$, where $\mu_b$ is the branch-specific mean and $\xi_{i,j}$ are i.i.d. zero-mean random variables with variance $\sigma^2 \approx 1$. Then the $\ell_1$ norm of each row of the sty up to an approximation error of order $\mathcal{O}(\sigma^3)$.

Figures (13)

  • Figure 1: Different pipelines for style-guided image synthesis: (a) style-related text-guided image synthesis; (b) training-based style-guided image synthesis (InstaStyle INSTASTYLE); (c) training-free style-guided image synthesis (StyleAligned Style_Aligned); and (d) our method, which simply leverages in-context learning in a pre-trained ReFlow inpainting model enhanced by dynamic semantic-style integration for robust fusion.
  • Figure 2: Examples of style-guided image synthesis using our method Sissi. Given a style reference, our approach generates stylized images conditioned on a content prompt, achieving exceptional quality and fidelity. Furthermore, it seamlessly extends to multiple style-guided synthesis and image editing tasks, demonstrating versatility and superior performance.
  • Figure 3: Overview of our framework's pipeline. Given a style image $I_s$, we concatenate it with a target mask and input the combined image to the generation network, leveraging its in-context style adaptation capability. The DSSI module adaptively balances attention between textual and visual style cues, yielding outputs that incorporate target style patterns while preserving content semantics.
  • Figure 4: An illustration of semantic-style imbalance. Subfigure (a) shows a failure case from the baseline model, where the prompt dominates the output ($\mu_p\gg\mu_s$). In contrast, our approach achieves a more balanced generation where the style ($\mu_s$) and prompt ($\mu_p$) contributions are comparable.
  • Figure 5: An illustration depicting how noise introduced by masking the input propagates through logits, attention weights, and the output tensor. The Mean Absolute Error (MAE) quantifies the differences between results obtained with masking and the reference results. As the masked area increases, these differences become more pronounced.
  • ...and 8 more figures

Theorems & Definitions (3)

  • Proposition 1
  • Proposition 2
  • Proposition 3