Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace

TL;DR

BaTex tackles the inefficiency and limited prompt compatibility of prior personalized text-to-image methods by learning target embeddings within a low-dimensional textual subspace constructed from pre-trained vocabularies. By leveraging a self-expressiveness principle, BaTex represents any target embedding as a linear combination of basis embeddings and optimizes in this restricted subspace, improving text-to-image alignment while maintaining reconstruction quality. The method includes a rank-based basis selection strategy to ensure semantically relevant basis vectors and a theoretical result showing the update is equivalent to a projection with rank d1. Empirically, BaTex outperforms existing embedding-only approaches in text alignment and competes with model-fine-tuning methods, while offering faster convergence and robustness to initialization, making personalized generation more scalable and controllable.

Abstract

Personalized text-to-image generation has attracted unprecedented attention in the recent few years due to its unique capability of generating highly-personalized images via using the input concept dataset and novel textual prompt. However, previous methods solely focus on the performance of the reconstruction task, degrading its ability to combine with different textual prompt. Besides, optimizing in the high-dimensional embedding space usually leads to unnecessary time-consuming training process and slow convergence. To address these issues, we propose an efficient method to explore the target embedding in a textual subspace, drawing inspiration from the self-expressiveness property. Additionally, we propose an efficient selection strategy for determining the basis vectors of the textual subspace. The experimental evaluations demonstrate that the learned embedding can not only faithfully reconstruct input image, but also significantly improves its alignment with novel input textual prompt. Furthermore, we observe that optimizing in the textual subspace leads to an significant improvement of the robustness to the initial word, relaxing the constraint that requires users to input the most relevant initial word. Our method opens the door to more efficient representation learning for personalized text-to-image generation.

Figures (15)

• Figure 1: Method Comparison of Textual Inversion and BaTex. Red dashed area denotes the textual subspace $\mathbb S$.
• Figure 2: Visualization comparison of different methods.
• Figure 3: Concept composition of multiple learned embeddings. Bolded texts are additional text conditions.
• Figure 4: Qualitative results of BaTex. Bolded texts are additional text conditions.
• Figure 5: Additional qualitative results of BaTex. Bolded texts are additional text conditions.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2
• Theorem 3
• Proof 1
• Theorem 4
• Proof 2