YoChameleon: Personalized Vision and Language Generation

TL;DR

This work tackles the problem of personalizing large multimodal models that can both understand and generate images and text, using only 3–5 exemplar images per concept. It introduces Yo'Chameleon, a dual-prefix soft-prompt framework with soft-positive data and a self-prompting mechanism to balance text and image tasks while preserving general knowledge. A key finding is that soft-prompt tuning can match full-model fine-tuning performance with roughly 300 real images, and that adaptive soft-positive training with separate token sets for each modality yields strong, task-specific personalization without catastrophic forgetting. The approach offers a practical path toward user-specific, vision-language personalization in LMMs and shows competitive gains over baselines like Chameleon and GPT-4o in several personalized generation tasks.

Abstract

Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous work has explored personalization for text generation, yet it remains unclear how these methods can be adapted to new modalities, such as image generation. In this paper, we introduce Yo'Chameleon, the first attempt to study personalization for large multimodal models. Given 3-5 images of a particular concept, Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information to (i) answer questions about the subject and (ii) recreate pixel-level details to produce images of the subject in new contexts. Yo'Chameleon is trained with (i) a self-prompting optimization mechanism to balance performance across multiple modalities, and (ii) a ``soft-positive" image generation approach to enhance image quality in a few-shot setting.

Figures (13)

• Figure 1: Comparisons with Chameleon chameleon and GPT-4o gpt4o using personalized image/text prompts. Our approach achieves significantly improved personalized image generation capabilities.
• Figure 2: Image Reconstruction. The generated image, conditioned on a personalized prompt, is compared with the ground truth image to calculate the image reconstruction loss.
• Figure 3: "Soft positive" images. Retrieved images are ranked according to their similarity to positive images using CLIP image similarity scores. Images that are more similar to the actual positive images are described with more latent tokens (i.e., more details).
• Figure 3: Soft-Prompt Tuning vs. Full-Model Fine-Tuning. Overall, soft-prompt tuning matches the performance of full-model fine-tuning for personalized abilities while retaining the original model's general capabilities.
• Figure 4: Optimized tokens for one task cannot effectively perform another, and simply training on a mixture of data yields suboptimal performance across tasks. We propose a self-prompting approach, where the model predicts which task to perform first, achieving the best of both worlds. (Input images are given in Fig. \ref{['fig:feature-graphic']}).