Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Editability in Image Generation with Layer-wise Memory

Daneul Kim, Jaeah Lee, Jaesik Park

TL;DR

This work tackles the challenge of sequential image editing with rough masks by introducing a three-component framework: Layer-wise Memory to preserve editing history, Background Consistency Guidance (BCG) for stable background maintenance, and Multi-Query Disentangled Cross-Attention (MQD) for natural integration of new elements. The method stores latent representations and prompt embeddings across edits, enabling efficient, consistent updates via selective latent blending and cross-attention disentanglement, while a new Multi-Edit Benchmark evaluates semantic and spatial alignment across multiple edits. Quantitative and qualitative results demonstrate superior performance over editing and layout-to-image baselines, as well as favorable human judgments, particularly in maintaining background coherence and object relations through several iterations. This approach advances practical, interactive image editing by reducing user effort (rough masks suffice) and enabling robust multi-step scene modifications with context preservation. By combining memory, consistency guidance, and disentangled cross-attention, the framework achieves scalable, layer-wise editing suitable for complex tasks such as occluded object removal, depth-aware ordering, and progressive content insertion, with potential applications in design, advertising, and creative media generation.

Abstract

Most real-world image editing tasks require multiple sequential edits to achieve desired results. Current editing approaches, primarily designed for single-object modifications, struggle with sequential editing: especially with maintaining previous edits along with adapting new objects naturally into the existing content. These limitations significantly hinder complex editing scenarios where multiple objects need to be modified while preserving their contextual relationships. We address this fundamental challenge through two key proposals: enabling rough mask inputs that preserve existing content while naturally integrating new elements and supporting consistent editing across multiple modifications. Our framework achieves this through layer-wise memory, which stores latent representations and prompt embeddings from previous edits. We propose Background Consistency Guidance that leverages memorized latents to maintain scene coherence and Multi-Query Disentanglement in cross-attention that ensures natural adaptation to existing content. To evaluate our method, we present a new benchmark dataset incorporating semantic alignment metrics and interactive editing scenarios. Through comprehensive experiments, we demonstrate superior performance in iterative image editing tasks with minimal user effort, requiring only rough masks while maintaining high-quality results throughout multiple editing steps.

Improving Editability in Image Generation with Layer-wise Memory

TL;DR

This work tackles the challenge of sequential image editing with rough masks by introducing a three-component framework: Layer-wise Memory to preserve editing history, Background Consistency Guidance (BCG) for stable background maintenance, and Multi-Query Disentangled Cross-Attention (MQD) for natural integration of new elements. The method stores latent representations and prompt embeddings across edits, enabling efficient, consistent updates via selective latent blending and cross-attention disentanglement, while a new Multi-Edit Benchmark evaluates semantic and spatial alignment across multiple edits. Quantitative and qualitative results demonstrate superior performance over editing and layout-to-image baselines, as well as favorable human judgments, particularly in maintaining background coherence and object relations through several iterations. This approach advances practical, interactive image editing by reducing user effort (rough masks suffice) and enabling robust multi-step scene modifications with context preservation. By combining memory, consistency guidance, and disentangled cross-attention, the framework achieves scalable, layer-wise editing suitable for complex tasks such as occluded object removal, depth-aware ordering, and progressive content insertion, with potential applications in design, advertising, and creative media generation.

Abstract

Most real-world image editing tasks require multiple sequential edits to achieve desired results. Current editing approaches, primarily designed for single-object modifications, struggle with sequential editing: especially with maintaining previous edits along with adapting new objects naturally into the existing content. These limitations significantly hinder complex editing scenarios where multiple objects need to be modified while preserving their contextual relationships. We address this fundamental challenge through two key proposals: enabling rough mask inputs that preserve existing content while naturally integrating new elements and supporting consistent editing across multiple modifications. Our framework achieves this through layer-wise memory, which stores latent representations and prompt embeddings from previous edits. We propose Background Consistency Guidance that leverages memorized latents to maintain scene coherence and Multi-Query Disentanglement in cross-attention that ensures natural adaptation to existing content. To evaluate our method, we present a new benchmark dataset incorporating semantic alignment metrics and interactive editing scenarios. Through comprehensive experiments, we demonstrate superior performance in iterative image editing tasks with minimal user effort, requiring only rough masks while maintaining high-quality results throughout multiple editing steps.
Paper Structure (42 sections, 10 equations, 18 figures, 7 tables)

This paper contains 42 sections, 10 equations, 18 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Layer-wise Memory
  6. Background Consistency Guidance
  7. Multi-Query Disentangled Cross-Attention
  8. Cross-Attention for Current Object
  9. Disentangled Attention for Previous Mask Orders
  10. Final Latent Update
  11. Improving Editability
  12. Multi-Edit Benchmark
  13. Experiment
  14. Implementation Details
  15. Quantitative Result
  16. ...and 27 more sections

Figures (18)

  • Figure 1: Overview. Our framework enables the interactive generation of images with enhanced control but in a simple manner, by rough mask and prompt, through iterative scene editing. We utilize the background scene generated by our framework to edit in HD Painter manukyan2023hd or Blended Latent Diffusion (BLD) avrahami2023blended for comparison and commercial products like Photoshop photoshop and Pincel pincel.
  • Figure 2: Overview. (a) The left denotes an illustration of how Multi Query Disentanglement is performed in the cross-attention layer. (b) The upper right figure shows Background Consistency Guidance with recalled latents, conducting latent blending with the saved latents. (c) The right below shows the layer-wise memory, saving the previous editing steps' latents, masks, and prompt embeddings.
  • Figure 3: Overview of our proposed Multi-Edit Benchmark for evaluation of iterative editing scenario. (a) explains the dataset generation pipeline through GPT-4 API, and (b) explains the evaluation methodology in visual alignment using CLIP and semantic alignment using LLaVa for single-image and in a layer-wise manner.
  • Figure 4: Qualitative comparison on the effect of Query Disentanglement (QD).
  • Figure 5: Comparison in image editing capability with latest image editing models.manukyan2023hdavrahami2023blended Note that the initial image is generated by our framework, which is equivalent to PixArt-$\alpha$chen2023pixartalpha with no mask input.
  • ...and 13 more figures