Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing

TL;DR

This work targets the common problem in diffusion-based text-driven editing where preserving the source structure is difficult with DDS. It introduces Contrastive Denoising Score (CDS), which embeds a CUT-inspired PatchNCE loss on latent diffusion model self-attention features into the DDS framework, enabling zero-shot, structure-aware edits and even extending to NeRF editing. The approach provides a simple, training-free regularizer that balances semantic change with structural fidelity, showing qualitative and quantitative improvements over baselines. By leveraging intermediate self-attention representations, CDS achieves robust 2D edits and extends the methodology to 3D NeRF representations, broadening the practical impact of text-guided image and scene editing.

Abstract

With the remarkable advent of text-to-image diffusion models, image editing methods have become more diverse and continue to evolve. A promising recent approach in this realm is Delta Denoising Score (DDS) - an image editing technique based on Score Distillation Sampling (SDS) framework that leverages the rich generative prior of text-to-image diffusion models. However, relying solely on the difference between scoring functions is insufficient for preserving specific structural elements from the original image, a crucial aspect of image editing. To address this, here we present an embarrassingly simple yet very powerful modification of DDS, called Contrastive Denoising Score (CDS), for latent diffusion models (LDM). Inspired by the similarities and differences between DDS and the contrastive learning for unpaired image-to-image translation(CUT), we introduce a straightforward approach using CUT loss within the DDS framework. Rather than employing auxiliary networks as in the original CUT approach, we leverage the intermediate features of LDM, specifically those from the self-attention layers, which possesses rich spatial information. Our approach enables zero-shot image-to-image translation and neural radiance field (NeRF) editing, achieving structural correspondence between the input and output while maintaining content controllability. Qualitative results and comparisons demonstrates the effectiveness of our proposed method. Project page: https://hyelinnam.github.io/CDS/

Figures (14)

• Figure 1: Text-guided Image Editing results via Contrastive Denoising Score. CDS successfully translates source images with a well-balanced interplay between maintaining the structural elements of the source image and transforming the content in alignment with the target text prompt.
• Figure 2: Overall pipeline of CDS. We extract the intermediate features of the self-attention layers and calculate $\ell_{con}$. This loss enables us to regulate structural consistency and generate reliable images.
• Figure 3: Comparison with baseline models. CDS demonstrates outstanding performance in effectively regulating structural consistency.
• Figure 4: Sample results of the cat2dog task from DDS and CDS.
• Figure 5: Gradient visualization on DDS and CDS.