Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models

Xin Jin, Yichuan Zhong, Yapeng Tian

TL;DR

TP-Blend is presented, a lightweight training-free framework that receives two separate textual prompts, one specifying a blend object and the other defining a target style, and injects both into a single denoising trajectory, driven by two complementary attention processors.

Abstract

Current text-conditioned diffusion editors handle single object replacement well but struggle when a new object and a new style must be introduced simultaneously. We present Twin-Prompt Attention Blend (TP-Blend), a lightweight training-free framework that receives two separate textual prompts, one specifying a blend object and the other defining a target style, and injects both into a single denoising trajectory. TP-Blend is driven by two complementary attention processors. Cross-Attention Object Fusion (CAOF) first averages head-wise attention to locate spatial tokens that respond strongly to either prompt, then solves an entropy-regularised optimal transport problem that reassigns complete multi-head feature vectors to those positions. CAOF updates feature vectors at the full combined dimensionality of all heads (e.g., 640 dimensions in SD-XL), preserving rich cross-head correlations while keeping memory low. Self-Attention Style Fusion (SASF) injects style at every self-attention layer through Detail-Sensitive Instance Normalization. A lightweight one-dimensional Gaussian filter separates low- and high-frequency components; only the high-frequency residual is blended back, imprinting brush-stroke-level texture without disrupting global geometry. SASF further swaps the Key and Value matrices with those derived from the style prompt, enforcing context-aware texture modulation that remains independent of object fusion. Extensive experiments show that TP-Blend produces high-resolution, photo-realistic edits with precise control over both content and appearance, surpassing recent baselines in quantitative fidelity, perceptual quality, and inference speed.

TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models

TL;DR

TP-Blend is presented, a lightweight training-free framework that receives two separate textual prompts, one specifying a blend object and the other defining a target style, and injects both into a single denoising trajectory, driven by two complementary attention processors.

Abstract

Current text-conditioned diffusion editors handle single object replacement well but struggle when a new object and a new style must be introduced simultaneously. We present Twin-Prompt Attention Blend (TP-Blend), a lightweight training-free framework that receives two separate textual prompts, one specifying a blend object and the other defining a target style, and injects both into a single denoising trajectory. TP-Blend is driven by two complementary attention processors. Cross-Attention Object Fusion (CAOF) first averages head-wise attention to locate spatial tokens that respond strongly to either prompt, then solves an entropy-regularised optimal transport problem that reassigns complete multi-head feature vectors to those positions. CAOF updates feature vectors at the full combined dimensionality of all heads (e.g., 640 dimensions in SD-XL), preserving rich cross-head correlations while keeping memory low. Self-Attention Style Fusion (SASF) injects style at every self-attention layer through Detail-Sensitive Instance Normalization. A lightweight one-dimensional Gaussian filter separates low- and high-frequency components; only the high-frequency residual is blended back, imprinting brush-stroke-level texture without disrupting global geometry. SASF further swaps the Key and Value matrices with those derived from the style prompt, enforcing context-aware texture modulation that remains independent of object fusion. Extensive experiments show that TP-Blend produces high-resolution, photo-realistic edits with precise control over both content and appearance, surpassing recent baselines in quantitative fidelity, perceptual quality, and inference speed.
Paper Structure (32 sections, 21 equations, 13 figures, 4 tables)

This paper contains 32 sections, 21 equations, 13 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Main Contributions.
  3. Code Availability.
  4. Related Work
  5. Proposed Method
  6. Preliminaries
  7. Twin-Prompt Attention Blend
  8. Cross-Attention Object Fusion
  9. Identifying Significant Positions in Cross-Attention Maps.
  10. Blending Feature Embeddings in Reshaped Cross-Attention Outputs.
  11. Formulating the Optimal Transport Problem.
  12. Solving the Optimal Transport Problem with the Sinkhorn Algorithm.
  13. Self-Attention Style Fusion
  14. Detail-Sensitive Instance Normalization.
  15. Key/Value Substitution.
  16. ...and 17 more sections

Figures (13)

  • Figure 1: Demonstration of our method's capabilities. Row 1: Original object "Knight" is replaced by "Leonardo DiCaprio", blended with "Batman", and styled with "Pop Art". Row 2: Original object "Robot" is replaced by "Knight", blended with "Thanos", and styled with "Cyberpunk Style". Row 3: Original object "Cat" is replaced by "Dog", blended with "Horse", and styled with "Oil painting". Row 4: Original object "Chameleon" is replaced by "Dinosaur", blended with "Fish", and styled with "Oil painting".
  • Figure 2: Flowchart of TP-Blend, integrating object replacement, blending, and style transfer within the diffusion process. In this example, the original object "Knight" is replaced by "Leonardo DiCaprio", blended with "Captain Jack Sparrow", and styled with a "Charcoal Drawing" effect.
  • Figure 3: CAOF Object Blending across different sets. Row 1: Original object “alpaca” is replaced by “puppy” and blended with “monkey”. Row 2: Original “apple” is replaced by “orange” and blended with “tomato”. Row 3: Original “frog” is replaced by “chameleon” and blended with “dinosaur”. Row 4: Original “truck” is replaced by “jeep” and blended with “ambulance”.
  • Figure 4: CAOF Flowchart: Cross-Attention Object Fusion merges the blend object's features into the replaced object by identifying key spatial positions in the attention maps and applying an optimal transport framework for coherent morphological transitions.
  • Figure 5: Object blending enhanced with various artistic styles. Style 1: Pixel Art; Style 2: Chocolate; Style 3: Charcoal Drawing; Style 4: Oil Painting.
  • ...and 8 more figures