Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEGA: Instructing Text-to-Image Models using Semantic Guidance

Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, Kristian Kersting

TL;DR

Problem: one-shot diffusion prompts lack precise semantic control, causing unstable edits. Approach: SEGA extracts concept-conditioned directions from the model's noise estimates and combines them with classifier-free guidance without retraining or architectural changes. Contributions: formalizes semantic space, proves robustness, uniqueness, monotonicity, and isolation of concept directions, and demonstrates broad empirical gains across latent and pixel diffusion models, multi-concept edits, style transfer, and safety mitigation. Impact: enables versatile, interactive image editing with strong fidelity and cross-model applicability, informing future work on latent representations and bias-aware editing.

Abstract

Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user's intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, changes in composition and style, as well as optimizing the overall artistic conception. We demonstrate SEGA's effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility, flexibility, and improvements over existing methods.

SEGA: Instructing Text-to-Image Models using Semantic Guidance

TL;DR

Problem: one-shot diffusion prompts lack precise semantic control, causing unstable edits. Approach: SEGA extracts concept-conditioned directions from the model's noise estimates and combines them with classifier-free guidance without retraining or architectural changes. Contributions: formalizes semantic space, proves robustness, uniqueness, monotonicity, and isolation of concept directions, and demonstrates broad empirical gains across latent and pixel diffusion models, multi-concept edits, style transfer, and safety mitigation. Impact: enables versatile, interactive image editing with strong fidelity and cross-model applicability, informing future work on latent representations and bias-aware editing.

Abstract

Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user's intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, changes in composition and style, as well as optimizing the overall artistic conception. We demonstrate SEGA's effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility, flexibility, and improvements over existing methods.
Paper Structure (34 sections, 10 equations, 25 figures, 4 tables, 1 algorithm)

This paper contains 34 sections, 10 equations, 25 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Semantic Guidance
  4. Guided Diffusion
  5. Semantic Guidance on Concepts
  6. Properties of Semantic Space
  7. Experimental Evaluation
  8. Broader Impact on Society
  9. Conclusions
  10. Acknowledgments
  11. Implementation of Sega
  12. Numerical Properties of Noise Estimates
  13. Further intuition and ablations on hyper-parameters
  14. Scale $s_e$.
  15. Threshold $\lambda$.
  16. ...and 19 more sections

Figures (25)

  • Figure 1: Semantic guidance (SEGA) applied to the image 'a portrait of a king' (Best viewed in color)
  • Figure 2: Numerical intuition of semantic guidance. The difference between the concept-conditioned and unconditioned estimates is first scaled. Subsequently, the tail values represent the dimensions of the specified concept. Distribution plots calculated using kernel-density estimates with Gaussian smoothing.
  • Figure 3: Robustness, uniqueness and monotonicity of Sega guidance vectors. In a) and b) the top row depicts the unchanged image, bottom row depicts the ones guided towards 'glasses'. (Best viewed in color)
  • Figure 4: Successive combination of concepts. From left to right an new concept is added each image. Concepts do not interfere with each other and only change the relevant portion of the image. (Best viewed in color)
  • Figure 5: Examples from our empirical evaluation benchmark, showing the 10 attributes edited with Sega. Original and edited images are evaluated by human users one feature at a time. (Best viewed in color)
  • ...and 20 more figures