TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization

TL;DR

TALE tackles cross-domain image composition without training or finetuning diffusion models. It leverages latent-space operations through Adaptive Latent Manipulation to initialize compositing latents at a strategically chosen timestep $T'<T$ and Adaptive Latent Normalization to harmonize content and style, followed by Energy-guided Latent Optimization that uses a CLIP-based energy term (and optionally a style term via Gram matrices) to align the result with the target prompt. The approach demonstrates state-of-the-art performance among training-free methods and competitive results against trained baselines across photorealistic and artistic domains, supported by quantitative and user-study evaluations. The methodology enables robust preservation of object identity and seamless background integration, highlighting the practical potential of training-free diffusion-guided editing for diverse domain adaptation.

Abstract

We present TALE, a novel training-free framework harnessing the generative capabilities of text-to-image diffusion models to address the cross-domain image composition task that focuses on flawlessly incorporating user-specified objects into a designated visual contexts regardless of domain disparity. Previous methods often involve either training auxiliary networks or finetuning diffusion models on customized datasets, which are expensive and may undermine the robust textual and visual priors of pre-trained diffusion models. Some recent works attempt to break the barrier by proposing training-free workarounds that rely on manipulating attention maps to tame the denoising process implicitly. However, composing via attention maps does not necessarily yield desired compositional outcomes. These approaches could only retain some semantic information and usually fall short in preserving identity characteristics of input objects or exhibit limited background-object style adaptation in generated images. In contrast, TALE is a novel method that operates directly on latent space to provide explicit and effective guidance for the composition process to resolve these problems. Specifically, we equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. The former formulates noisy latents conducive to initiating and steering the composition process by directly leveraging background and foreground latents at corresponding timesteps, and the latter exploits designated energy functions to further optimize intermediate latents conforming to specific conditions that complement the former to generate desired final results. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition across various photorealistic and artistic domains.

Figures (7)

• Figure 1: Illustration for the overall framework of TALE. First, the background latent $\mathbf{z}_0^{bg}$ and foreground latent $\mathbf{z}_0^{fg}$ are inverted into their respective noisy correspondences $\mathbf{z}_T^{bg}$ and $\mathbf{z}_T^{fg}$. Then, for selected timestep $T'$, we initiate the composition process by incorporating $\mathbf{z}_{T'}^{bg}$ and $\mathbf{z}_{T'}^{fg}$ via Selective Initiation (Section \ref{['adap']}). In subsequent timesteps $t\in[T'-\tau,T')$, the intermediate latent $\mathbf{z}_{t}^{res}$ is progressively refined through the sequential application of Adaptive Latent Normalization (Section \ref{['adap']}) and Energy-guided Latent Optimization (Section \ref{['energy']}), ultimately yielding the desired composited result $\mathbf{z}_0^{res}$.
• Figure 2: Our proposed TALE is robust against identity feature loss and noticeable artifacts indicating domain style disparity compared to TF-ICON.
• Figure 3: Qualitative comparison of TALE with prior SOTA and concurrent works in cross-domain image-guided composition. From top to bottom are representative results for compositing between real and watercolor, oil painting, comic, photorealism, sketching, and cartoon animation domains. Zoom-in for details.
• Figure 4: Quantitative comparison of TALE with prior SOTA works in cross-domain composition on the baseline benchmark with sketching, oil painting, and cartoon animation domains, and on the extended benchmark containing mixture of other domains such as comic and watercolor painting.
• Figure 5: User preference of TALE over prior works.