LAMS-Edit: Latent and Attention Mixing with Schedulers for Improved Content Preservation in Diffusion-Based Image and Style Editing

TL;DR

LAMS-Edit addresses the persistent challenge of preserving content structure while applying edits in diffusion-based image editing by leveraging the intermediate states $\{\mathbf{z}_t^*\}$ and $\{\mathbf{A}_t^*\}$ from DDIM inversion. It blends inversion-derived signals with generation-time representations via schedulers, performing separate latent and attention mixing governed by decaying weights, and integrates with Prompt-to-Prompt for targeted edits. The framework also supports region-specific editing with SAM masks and enables style transfer through LoRA in a tuning-free manner. Across SD1-5 and Anything-V4, LAMS-Edit demonstrates improved fidelity-editability trade-offs, with ablations confirming the contributions of latent/attention mixing and the schedulers to preserving identity and structure during edits, and is capable of simultaneous content edits and style adaptation with practical masking and masking-aware constraints.

Abstract

Text-to-Image editing using diffusion models faces challenges in balancing content preservation with edit application and handling real-image editing. To address these, we propose LAMS-Edit, leveraging intermediate states from the inversion process--an essential step in real-image editing--during edited image generation. Specifically, latent representations and attention maps from both processes are combined at each step using weighted interpolation, controlled by a scheduler. This technique, Latent and Attention Mixing with Schedulers (LAMS), integrates with Prompt-to-Prompt (P2P) to form LAMS-Edit--an extensible framework that supports precise editing with region masks and enables style transfer via LoRA. Extensive experiments demonstrate that LAMS-Edit effectively balances content preservation and edit application.

Figures (21)

• Figure 1: Overview of LAMS-Edit's capabilities. LAMS-Edit enhances structure and content preservation in T2I editing and style transfer.
• Figure 2: Overview of LAMS-Edit. Given an input image $\mathbf{x}_0$, DDIM inversion computes the initial latent representation $\mathbf{z}_T^*$, along with intermediate latent representations $\mathbf{z}^*$ and attention maps $\mathbf{A}^*$. These are then utilized by LAMS in the generation process, which is integrated with P2P to produce the edited image $\mathbf{\hat{x}}_0$. Optionally, a mask $\mathbf{M}$ generated by SAM kirillov2023sam can be applied to improve the spatial precision of edits. Additionally, LoRA-based style transfer can be applied simultaneously.
• Figure 3: Overview of LAMS. At each inversion step, the latent and attention maps are extracted and mixed with their counterparts in the generation process using independent schedulers. The mixing procedures are computed as shown in Eq. \ref{['eq:lams_latent_mixing']} and \ref{['eq:lams_attention_mixing']}. For clarity in the diagram, we denote $\bar{w}^\mathbf{z}_t = 1-w^\mathbf{z}_t$ and $\bar{w}^\mathbf{A}_t = 1-w^\mathbf{A}_t$, and omit P2P and LoRA for simplicity.
• Figure 4: Fidelity-editability trade-off of the image editing methods. Closer proximity to the lower right indicates a better balance.
• Figure 5: Results of different image editing methods for different editing scenarios.