TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space

TL;DR

TokenVerse tackles multi-concept personalized image generation by learning per-token modulation directions in the modulation space $\mathcal{M}^+$ of a diffusion Transformer. A lightweight Concept-Mod predicts token-wise directions, enabling unsupervised disentanglement of multiple concepts from multiple images and plug-and-play composition without segmentation masks. The method uses a two-stage, per-block optimization with a concept isolation loss to minimize cross-concept interference, achieving high concept preservation and prompt fidelity on DreamBench++-style benchmarks and user studies. This approach broadens controllable content creation to objects, poses, materials, and lighting, with strong implications for storytelling and customized visual generation.

Abstract

We present TokenVerse -- a method for multi-concept personalization, leveraging a pre-trained text-to-image diffusion model. Our framework can disentangle complex visual elements and attributes from as little as a single image, while enabling seamless plug-and-play generation of combinations of concepts extracted from multiple images. As opposed to existing works, TokenVerse can handle multiple images with multiple concepts each, and supports a wide-range of concepts, including objects, accessories, materials, pose, and lighting. Our work exploits a DiT-based text-to-image model, in which the input text affects the generation through both attention and modulation (shift and scale). We observe that the modulation space is semantic and enables localized control over complex concepts. Building on this insight, we devise an optimization-based framework that takes as input an image and a text description, and finds for each word a distinct direction in the modulation space. These directions can then be used to generate new images that combine the learned concepts in a desired configuration. We demonstrate the effectiveness of TokenVerse in challenging personalization settings, and showcase its advantages over existing methods. project's webpage in https://token-verse.github.io/

Figures (21)

• Figure 1: TokenVerse extracts distinct complex visual concepts from a set of concept images (top), and allows users to generate images that depict these concepts in novel versatile compositions (bottom row). Our framework independently processes each concept image, and learns to disentangle its concepts based solely on an accompanying caption, without any additional supervision or masks. This is achieved by learning a personalized representation for each token in the source caption. Our personalized text tokens, extracted from multiple images, are then flexibly incorporated into new text prompts (colored words) to generate novel creative images.
• Figure 2: Directions in the global modulation space ($\mathcal{M}$) and our per-token modulation space ($\mathcal{M}^+$). Given a generated image (top row), we modify it using text-driven directions in both $\mathcal{M}$ and $\mathcal{M}^+$ spaces. (a) Adding a direction to the vector that is used to modulate all the text and image tokens (i.e. a direction in the space $\mathcal{M}$) can be used to effectively modify desired concepts in the generated image. Yet, this often results in non-local changes that also affect other concepts in the generated image. (b) Adding a direction only to the modulation vector of a specific text token, like "dog" or "ball" (i.e. a direction in the space $\mathcal{M}^+$) leads to a localized modification that mostly affects the concept of interest.
• Figure 3: TokenVerse overview. (a) A pre-trained text-to-image DiT model processes both image and text tokens via a series of DiT blocks. Each block consists of modulation, attention and feed-forward modules. We focus on the modulation block, in which the tokens are modulated via a vector $y$, which is derived from a pooled text embedding. (b) Given a concept image and its corresponding caption, TokenVerse learns a personalized modulation vector offset $\Delta$ for each text token. These offsets represent personalized directions in the modulation space and are learned using a simple reconstruction objective. (c) At inference, the pre-learned direction vectors are used to modulate the text tokens, enabling the injection of personalized concepts into the generated images.
• Figure 4: Concept isolation loss. When training Concept-Mod we apply an additional concept isolation loss in 50% of the training steps. This loss encourages learning directions that do not interfere with other images by enforcing that the parts in the image that should not be affected by the directions remain similar.
• Figure 5: Qualitative results. Each row begins with a bank of four source images, from which our method independently extracts concepts. To the right, three generated images are shown, demonstrating the seamless combination of these concepts into new, coherent outputs.