Customization Assistant for Text-to-image Generation

TL;DR

This work tackles the challenge of customizing pre trained text to image generation for novel concepts without test time fine tuning. It introduces CAFE, a tuning free customization assistant that leverages a multi modal large language model to interpret prompts and guide a diffusion generator via cross attention, while also producing natural language explanations for generated content. A novel training strategy and dataset construction pipeline enable scalable self supervised training, including Self improvement via Distillation to generate more data without human labeling. Experiments across object and human domains show competitive quantitative performance, fast generation times, and improved user interaction through explanations, highlighting the practical impact of tuning free, interactive customization for open domain image synthesis.

Abstract

Customizing pre-trained text-to-image generation model has attracted massive research interest recently, due to its huge potential in real-world applications. Although existing methods are able to generate creative content for a novel concept contained in single user-input image, their capability are still far from perfection. Specifically, most existing methods require fine-tuning the generative model on testing images. Some existing methods do not require fine-tuning, while their performance are unsatisfactory. Furthermore, the interaction between users and models are still limited to directive and descriptive prompts such as instructions and captions. In this work, we build a customization assistant based on pre-trained large language model and diffusion model, which can not only perform customized generation in a tuning-free manner, but also enable more user-friendly interactions: users can chat with the assistant and input either ambiguous text or clear instruction. Specifically, we propose a new framework consists of a new model design and a novel training strategy. The resulting assistant can perform customized generation in 2-5 seconds without any test time fine-tuning. Extensive experiments are conducted, competitive results have been obtained across different domains, illustrating the effectiveness of the proposed method.

Figures (14)

• Figure 1: Generated example from the proposed CAFE. CAFE can perform customized generation based on the user provided image in a tuning-free manner. It outputs creative images along with text explanation and elaboration.
• Figure 2: Illustration of our model architecture, where the modules to be fine-tuned are indicated by flame icons.
• Figure 3: Our method enables tuning-free generation conditioned on multiple images. Details captured by $\mathcal{E}({\mathbf{x}})$ can be seamlessly combined with semantic $\mathcal{F}({\mathbf{w}})$.
• Figure 4: Two examples from our dataset, each sample contains four elements $({\mathbf{x}}, {\mathbf{y}}, \tilde{{\mathbf{x}}}, \tilde{{\mathbf{y}}})$.
• Figure 5: We can generate high quality training data efficiently with \ref{['eq:generation']}. The semantic and identity can also be easily controlled through single hyper-parameter $\alpha$.