Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models

Mor Ventura, Michael Toker, Or Patashnik, Yonatan Belinkov, Roi Reichart

TL;DR

DeLeaker introduces a lightweight, inference-time method that directly modulates attention in diffusion-based T2I models to suppress semantic leakage between entities. By automatically extracting entity masks from early cross-modal attention and applying targeted cross-entity suppression alongside self-identity strengthening, it mitigates leakage without external guidance and preserves image fidelity. The work contributes SLIM, the first large-scale dataset dedicated to semantic leakage in images, plus an automated evaluation framework validated by human judgments, and demonstrates superior leakage mitigation on FLUX and SANA models with robust ablations. Overall, DeLeaker establishes attention-control as a practical path toward semantically precise T2I generation and provides a rigorous benchmark for systematic evaluation.

Abstract

Text-to-Image (T2I) models have advanced rapidly, yet they remain vulnerable to semantic leakage, the unintended transfer of semantically related features between distinct entities. Existing mitigation strategies are often optimization-based or dependent on external inputs. We introduce DeLeaker, a lightweight, optimization-free inference-time approach that mitigates leakage by directly intervening on the model's attention maps. Throughout the diffusion process, DeLeaker dynamically reweights attention maps to suppress excessive cross-entity interactions while strengthening the identity of each entity. To support systematic evaluation, we introduce SLIM (Semantic Leakage in IMages), the first dataset dedicated to semantic leakage, comprising 1,130 human-verified samples spanning diverse scenarios, together with a novel automatic evaluation framework. Experiments demonstrate that DeLeaker consistently outperforms all baselines, even when they are provided with external information, achieving effective leakage mitigation without compromising fidelity or quality. These results underscore the value of attention control and pave the way for more semantically precise T2I models.

DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models

TL;DR

DeLeaker introduces a lightweight, inference-time method that directly modulates attention in diffusion-based T2I models to suppress semantic leakage between entities. By automatically extracting entity masks from early cross-modal attention and applying targeted cross-entity suppression alongside self-identity strengthening, it mitigates leakage without external guidance and preserves image fidelity. The work contributes SLIM, the first large-scale dataset dedicated to semantic leakage in images, plus an automated evaluation framework validated by human judgments, and demonstrates superior leakage mitigation on FLUX and SANA models with robust ablations. Overall, DeLeaker establishes attention-control as a practical path toward semantically precise T2I generation and provides a rigorous benchmark for systematic evaluation.

Abstract

Text-to-Image (T2I) models have advanced rapidly, yet they remain vulnerable to semantic leakage, the unintended transfer of semantically related features between distinct entities. Existing mitigation strategies are often optimization-based or dependent on external inputs. We introduce DeLeaker, a lightweight, optimization-free inference-time approach that mitigates leakage by directly intervening on the model's attention maps. Throughout the diffusion process, DeLeaker dynamically reweights attention maps to suppress excessive cross-entity interactions while strengthening the identity of each entity. To support systematic evaluation, we introduce SLIM (Semantic Leakage in IMages), the first dataset dedicated to semantic leakage, comprising 1,130 human-verified samples spanning diverse scenarios, together with a novel automatic evaluation framework. Experiments demonstrate that DeLeaker consistently outperforms all baselines, even when they are provided with external information, achieving effective leakage mitigation without compromising fidelity or quality. These results underscore the value of attention control and pave the way for more semantically precise T2I models.
Paper Structure (55 sections, 11 equations, 22 figures, 24 tables)

This paper contains 55 sections, 11 equations, 22 figures, 24 tables.

Table of Contents

  1. Introduction
  2. DeLeaker
  3. Attention-Based Entity Masking
  4. Attention-based Leakage Mitigation
  5. Leakage Suppression.
  6. Strengthening Self-Identity Alignment.
  7. The SLIM Dataset: Semantic-Leakage in Images
  8. Large-Scale Generation & Dataset Design.
  9. Human-Guided Filtering of Semantic Leakage.
  10. Evaluation
  11. Automatic Evaluation for Leakage Mitigation.
  12. Preservation Metrics.
  13. Human Assessment of Leakage Mitigation (User Study).
  14. Experimental Setup
  15. Base DiT T2I Models.
  16. ...and 40 more sections

Figures (22)

  • Figure 1: DeLeaker Qualitative Examples. Top: DeLeaker ; Bottom: original outputs. Red arrows mark features affected by semantic leakage. Examples cover five subsets of the SLIM dataset (§\ref{['sec:dataset']}).
  • Figure 2: DeLeaker Scheme. Our method applies attention-based interventions during the diffusion process: (A) automatically extracting entity-specific masks from early image-text attention maps; (B) suppressing cross-entity leakage by suppressing attention across entities in both image-text and image-image interactions; and (C) strengthening self-identity by increasing attention from each entity’s text tokens to its own image tokens. The attention map legend (left) shows how entities interact, where colors denote different interaction regions. The final output (right) presents the image output with DeLeaker compared to the original image, when DeLeaker is not applied.
  • Figure 2: DeLeaker Ablation Study. Configurations are divided into two types: (1) W/O rows (top four) represent the removal/addition of a specific component, while (2) Only rows (bottom three) isolate each component independently. Ratios are reported relative to the full DeLeaker scores baseline, with values closer to 1.0 indicating similarity. Darker hues indicate stronger contributions, color-coded as positive and negative. Signs indicate attention suppression (-) or strengthening (+).
  • Figure 3: Our Automatic Evaluation Framework for Assessing Semantic Leakage Mitigation. The framework consists of three main steps: (1) visual difference extraction, (2) typicality assessment, and (3) comparative ranking. Step 1 is divided into two parts: one based solely on the input prompt (top) and the other employs reference images generated for each entity (bottom). The VLM generates and then merges two independent descriptions into a unified description. Step 2 consists of four typicality questions, one for each entity in each image, guided by the unified differences identified in Step 1. Step 3 employs the outputs of Step 2 to compare both images. It produces a classification indicating the preferred image (Image 1 or Image 2) and the magnitude of change (minor or major).
  • Figure 4: Qualitative comparison across baselines (columns) and three examples (rows).
  • ...and 17 more figures