Edge-Aware Image Manipulation via Diffusion Models with a Novel Structure-Preservation Loss

TL;DR

The paper tackles the challenge of preserving pixel-level edge structures in latent-diffusion-model (LDM) image editing. It introduces Structure Preservation Loss (SPL), built on a Local Linear Model, to explicitly enforce structural fidelity during a training-free diffusion edit, aided by a gradient-descent denoising step and a post-processing refinement. A Color Preservation Loss (L_CPL) and a text-driven cross-attention mask upsampling scheme provide additional local control, enabling precise localized edits. Empirical results on PIE-Bench and AnyEdit show state-of-the-art structure preservation with competitive prompt fidelity, supported by extensive ablations and cost analyses that highlight practical efficiency and broad applicability across backbones.

Abstract

Recent advances in image editing leverage latent diffusion models (LDMs) for versatile, text-prompt-driven edits across diverse tasks. Yet, maintaining pixel-level edge structures-crucial for tasks such as photorealistic style transfer or image tone adjustment-remains as a challenge for latent-diffusion-based editing. To overcome this limitation, we propose a novel Structure Preservation Loss (SPL) that leverages local linear models to quantify structural differences between input and edited images. Our training-free approach integrates SPL directly into the diffusion model's generative process to ensure structural fidelity. This core mechanism is complemented by a post-processing step to mitigate LDM decoding distortions, a masking strategy for precise edit localization, and a color preservation loss to preserve hues in unedited areas. Experiments confirm SPL enhances structural fidelity, delivering state-of-the-art performance in latent-diffusion-based image editing. Our code will be publicly released at https://github.com/gongms00/SPL.

Figures (26)

• Figure 1: Structure-preserving edit. Our method achieves pixel-level structural fidelity without compromising the intended edit.
• Figure 1: Qualitative comparison on photorealistic style transfer. Given a content image (a) and a style image (b), our method effectively transfers the sunrise style, producing a naturally stylized result (d). In contrast, PCAKD (c) fails to transfer this style effectively.
• Figure 2: Motivation of the structure preservation loss. (Fig 2-1) An edited image may contain both structure-preserving (marked in blue) and structure-breaking regions (marked in orange). Our approach is motivated by the local linear model's ability to analyze these regions on a local window-by-window basis. (Left) When structure is preserved, the model finds an accurate linear fit, resulting in low error. (Right) When structure is broken, the model fails to find a good fit, producing a high error that signals the distortion. (Fig 2-2) Unidirectional structural comparison fails to fully capture mutual differences, motivating the bidirectional design of our structure preservation loss.
• Figure 2: Qualitative comparison on image harmonization. (a) Composed image with foreground mask (top-right). The Result from PCTNet (b) exhibits clear lighting inconsistencies. Our method (c) seamlessly blends the foreground and background.
• Figure 3: Structure-preserving denoising loop. At each denoising timestep $t$, we decode the predicted clean latent $\hat{z}_0^{(t)}$ to compute our structure preservation loss $\mathcal{L}_{\text{SPL}}$ in the image space. The resulting gradient is then used to update the latent, producing a corrected version $\tilde{z}$ that steers the generation trajectory to maintain structural fidelity for the subsequent denoising step.