Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps

Andrey Palaev, Adil Khan, Syed M. Ahsan Kazmi

TL;DR

This work tackles precise instance-level image editing in text-to-image diffusion without model retraining or auxiliary masks. It combines LLM-based object extraction with open-vocabulary detection to locate individual objects mentioned in prompts, then applies guidance derived from cross-attention maps and diffusion U-Net activations to manipulate positions while preserving appearances. The authors formulate position and preservation guidance terms and a total guidance objective to enable targeted edits, and validate the approach with qualitative comparisons and ablations. The results show coherent, instance-specific edits that outperform some prior methods in preserving detail and enabling precise repositioning, with practical implications for controllable image synthesis. Future work aims to reduce hyperparameter sensitivity and extend the technique to handling large objects more robustly, potentially by integrating Dynamic Prompt Learning to improve cross-attention fidelity.

Abstract

The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM

LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps

TL;DR

This work tackles precise instance-level image editing in text-to-image diffusion without model retraining or auxiliary masks. It combines LLM-based object extraction with open-vocabulary detection to locate individual objects mentioned in prompts, then applies guidance derived from cross-attention maps and diffusion U-Net activations to manipulate positions while preserving appearances. The authors formulate position and preservation guidance terms and a total guidance objective to enable targeted edits, and validate the approach with qualitative comparisons and ablations. The results show coherent, instance-specific edits that outperform some prior methods in preserving detail and enabling precise repositioning, with practical implications for controllable image synthesis. Future work aims to reduce hyperparameter sensitivity and extend the technique to handling large objects more robustly, potentially by integrating Dynamic Prompt Learning to improve cross-attention fidelity.

Abstract

The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM
Paper Structure (15 sections, 7 equations, 3 figures)

This paper contains 15 sections, 7 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background & Related work
  3. Diffusion models
  4. Guidance
  5. Diffusion-based Image editing
  6. Methodology
  7. LLM-based object detection
  8. Image editing with guidance
  9. Position
  10. Object preservation
  11. Total guidance term
  12. Results & Discussion
  13. Position
  14. Ablation study on different preservation terms
  15. Conclusion & Future Work

Figures (3)

  • Figure 1: Overview of our pipeline. Firstly, LLM parses the objects from the prompt. Then, an open-vocabulary object detector detects these objects on the image. Finally, the image is edited with the use of guidance. Note that this pipeline uses only pretrained models and does not require any training or fine-tuning.
  • Figure 2: Examples of the position manipulations. Coordinates shift is represented by (x, y)
  • Figure 3: Comparison of different preservation terms in our method. Coordinates shift is represented by (x, y).