Tuning-Free Image Customization with Image and Text Guidance

TL;DR

This work addresses the challenge of region-specific image customization without heavy tuning by blending text and image guidance within a diffusion-model framework. It introduces a collage-based inversion and a three-stream denoising pipeline that blends self-attention across reconstruction, text-guided, and edited streams to preserve reference subject features while enabling attribute edits driven by text prompts. Quantitative and qualitative evaluations show strong superiority over prior global/local editing methods and two-step pipelines, with high fidelity to the reference subject and good alignment to text descriptions. The approach enables practical applications in creative photography and graphic design, while acknowledging limitations in dynamic, non-rigid, or multi-view editing scenarios. Overall, the method offers a fast, versatile solution for precise, region-level image customization without fine-tuning of diffusion models.

Abstract

Despite significant advancements in image customization with diffusion models, current methods still have several limitations: 1) unintended changes in non-target areas when regenerating the entire image; 2) guidance solely by a reference image or text descriptions; and 3) time-consuming fine-tuning, which limits their practical application. In response, we introduce a tuning-free framework for simultaneous text-image-guided image customization, enabling precise editing of specific image regions within seconds. Our approach preserves the semantic features of the reference image subject while allowing modification of detailed attributes based on text descriptions. To achieve this, we propose an innovative attention blending strategy that blends self-attention features in the UNet decoder during the denoising process. To our knowledge, this is the first tuning-free method that concurrently utilizes text and image guidance for image customization in specific regions. Our approach outperforms previous methods in both human and quantitative evaluations, providing an efficient solution for various practical applications, such as image synthesis, design, and creative photography.

Figures (8)

• Figure 1: Performance overview of the proposed method in image customization: (a) The proposed method enables the generation of any subject depicted in the reference image within the designated image region to be edited. Additionally, it allows for modifying the generated subject's attributes based on the user's text description. (b) The versatility: our method can extend to scenarios involving multiple subjects from different reference images and multiple regions to be edited. (c) Cross-domain customization: driven by text, the proposed method can transform the subject in the reference image into a different domain, such as converting it into a cartoon style.
• Figure 2: The pipeline of the proposed method. Our method uses text descriptions $T$ and the reference image $I_r$ as guidance to customize the target region(s) of the image to be edited in a tuning-free manner. We employ blended self-attention instead of original self-attention injection throughout the denoising process, which allows us to retain (i) the generated subject features while achieving (ii) the text-driven capability for attributes modification.
• Figure 3: Reconstruction results. The first row shows the initial images used for inversion, the second row represents the image reconstruction results from DDIM song2020denoising, and the third row shows our reconstruction results. DDIM's results may distort when the object's material, lighting, or additional objects in the image are artificially altered. In contrast, our method consistently generates high-quality reconstructions, a critical aspect for image editing.
• Figure 4: Semantic information contained in different denoising steps. We observe that the layout is mainly formed in the early denoising process, while the generation of semantic information primarily begins in the latter stages. DINO caron2021dino score can reflect the richness of semantic information. Therefore, we perform the attention enhancement at this stage.
• Figure 5: Qualitative comparison with existing state-of-the-art methods. PBE yang2023pbe and AnyDoor chen2023anydoor are methods guided only by images, while BLD avrahami2023blended uses text as the only guidance. To evaluate the efficiency of our method, we set up an additional group of two-step methods, including first using image stitching and harmonization followed by text guided image editing (DCCF xue2022dccf + IP2P brooks2023instructpix2pix, MasaCtrl cao2023masactrl) and another method involving editing first and then harmonizing (IP2P brooks2023instructpix2pix + DCCF xue2022dccf). These methods can only focus on text or image, global or local editing. Our method outperforms all these methods and overcomes their limitations, achieving text and image guided local editing and generation.