HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation

TL;DR

A new hybrid framework called HybridBooth is presented, which merges the benefits of optimization-based and direct-regression methods and allows for effective and fast inversion of visual concepts into textual embedding, even from a single image, while maintaining the model's generalization capabilities.

Abstract

Recent advancements in text-to-image diffusion models have shown remarkable creative capabilities with textual prompts, but generating personalized instances based on specific subjects, known as subject-driven generation, remains challenging. To tackle this issue, we present a new hybrid framework called HybridBooth, which merges the benefits of optimization-based and direct-regression methods. HybridBooth operates in two stages: the Word Embedding Probe, which generates a robust initial word embedding using a fine-tuned encoder, and the Word Embedding Refinement, which further adapts the encoder to specific subject images by optimizing key parameters. This approach allows for effective and fast inversion of visual concepts into textual embedding, even from a single image, while maintaining the model's generalization capabilities.

Figures (13)

• Figure 1: We propose $\mathtt{HybridBooth}$ for efficient subject-driven generation. Unlike FastComposer xiao2023fastcomposer, which targets humans, it cannot affect the Bronze Mask (Top). Compared to optimization-based kumari2022customdiffusiondreambooth and direct-regression-based methods xiao2023fastcomposerwei2023elite, our framework generates diverse, precise images while preserving the subject's identity, requiring only $3\sim5$ optimization iterations.
• Figure 2: Comparing with previous frameworks, our method leverages a coarse-to-fine strategy, effectively integrating the advantages of both optimization- and regression-based methods. Consequently, we enable a fast prompt inversion with high-fidelity generation from just a single image.
• Figure 3: Showcasing our prompt inversion framework incorporating various conditions (depth, canny, keypoints) and community models.
• Figure 4: The framework of $\mathtt{HybridBooth}$ consists of two stages: (a) The first Probe Stage leverages a prompt regressor estimate the initial word embedding, acting as a probe for a rational word embedding; (b) The second Refinement Stage adapts the regressor to the subject by taking only 3-5 iterations to achieve efficient subject generation.
• Figure 5: Impact of hyper-parameters in the residual refinement stage (on the DreamBooth dataset dreambooth). Default parameters are marked with red circles. Custom Diffusion kumari2022customdiffusion serves as the reference. Prompt Fidelity and Subject Fidelity are CLIP-I and DINO-T, respectively (see \ref{['sec:exp_detail']}).