Novel Object Synthesis via Adaptive Text-Image Harmony

TL;DR

This paper introduces a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, and presents a novel similarity score function that maximizes the similarities between the generated object image and the input text/image.

Abstract

In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, \textit{i.e.}, often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects. First, we introduce a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration. Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar. Project page: https://xzr52.github.io/ATIH/.

Figures (27)

• Figure 1: We propose a straightforward yet powerful approach to generate combinational objects from a given object text-image pair for novel object synthesis. Our algorithm produces these combined object images using the central image and its surrounding text inputs, such as glass jar (image) and porcupine (text) in the left picture, and horse (image) and bald eagle (text) in the right picture.
• Figure 2: Imbalances between text and image in diffusion models. Using SDXL-Turbo sauer2023adversarial (left) and PnPinv Ju2024pnpinvers (right), the top pictures show a tendency for generated objects to align with textual content (green circles), while the bottom pictures tend to align with visual aspects (orange circles). In contrast, our approach achieves a more harmonious integration of both object text and image.
• Figure 3: Framework of our object synthesis incorporating a scale factor $\alpha$, an injection step $i$ and noise $\epsilon_t$ in the diffusion process. We design a balance loss for optimizing the noise $\epsilon_t$ to balance object editability and fidelity. Using the optimal noise $\epsilon_t$, we introduce an adaptive harmony mechanism to adjust $\alpha$ and $i$, balancing text (Peacock) and image (Rabbit) similarities.
• Figure 4: $I_{sim}$ and $T_{sim}$ with $\alpha\in [0,1.4]$.
• Figure 5: The adjusted process of our ATIH with three initial points and $\varepsilon=I_{\text{sim}}(\alpha) + k\cdot T_{\text{sim}}(\alpha)-F(\alpha)$.