PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery

TL;DR

PIXELS tackles exemplar-driven image editing by enabling region-wise, progressive edits through per-pixel control over denoising strength within off-the-shelf diffusion models, without any model training. It blends a source and one or more exemplars at inference time using a dynamic edit map, preserving text guidance and supporting multimodal prompts. A theoretical bound (Lemma 1) links denoising strength to latent-space adherence, guiding the trade-off between fidelity to the exemplar and realism of the edited image, and empirical results show improvements in FID, CLIP-I, and user preferences with minimal overhead. Overall, PIXELS broadens access to high-quality, professional-grade edits by reusing large pretrained diffusion models and enabling flexible, fine-grained editing across multiple exemplars.

Abstract

Recent advancements in language-guided diffusion models for image editing are often bottle-necked by cumbersome prompt engineering to precisely articulate desired changes. An intuitive alternative calls on guidance from in-the-wild image exemplars to help users bring their imagined edits to life. Contemporary exemplar-based editing methods shy away from leveraging the rich latent space learnt by pre-existing large text-to-image (TTI) models and fall back on training with curated objective functions to achieve the task. Though somewhat effective, this demands significant computational resources and lacks compatibility with diverse base models and arbitrary exemplar count. On further investigation, we also find that these techniques restrict user control to only applying uniform global changes over the entire edited region. In this paper, we introduce a novel framework for progressive exemplar-driven editing with off-the-shelf diffusion models, dubbed PIXELS, to enable customization by providing granular control over edits, allowing adjustments at the pixel or region level. Our method operates solely during inference to facilitate imitative editing, enabling users to draw inspiration from a dynamic number of reference images, or multimodal prompts, and progressively incorporate all the desired changes without retraining or fine-tuning existing TTI models. This capability of fine-grained control opens up a range of new possibilities, including selective modification of individual objects and specifying gradual spatial changes. We demonstrate that PIXELS delivers high-quality edits efficiently, leading to a notable improvement in quantitative metrics as well as human evaluation. By making high-quality image editing more accessible, PIXELS has the potential to enable professional-grade edits to a wider audience with the ease of using any open-source image generation model.

Figures (11)

• Figure 1: In-the-wild editing results produced by our method where users specify to-edit regions in the source image along with exemplars to inspire the edit. Our method changes different regions of the source by different amounts, according to a given non-binary edit map: the darker the region; more flexibility to adapt to the exemplar. This controllability allows us to create gradual spatial changes (e.g., forest-to-beach transition, top left) and transition across an edit realistically.
• Figure 2: Illustration of transition artifacts in the naive solution, generating unrealistic edits.
• Figure 3: Visualization of Algorithm \ref{['alg:inference']} Line \ref{['line14']} over time. Top: $z_1^{t} \odot mask_t$, regions copied from a noised version of the input. Bottom: $z^{t+1}_{mix} \odot (1 - mask_t)$, residue regions copied from the U-Net output in previous step. Note how the shifting mask at each timestep controls the inference process - darker the corresponding region in the edit map, the earlier it is copied from the residue. For ease of understanding, images are shown in the pixel space instead of the latent space.
• Figure 4: Qualitative comparisons. Our method can create realistic edits with high source and exemplar consistency.
• Figure 5: Results with multi-modal prompts. Text Prompts: “boat”, “watercolor style”, “low poly aesthetic, big monster”.