Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Yo'LLaVA: Your Personalized Language and Vision Assistant

Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, Yong Jae Lee

TL;DR

Yo'LLaVA tackles personalized multimodal conversation by embedding a user-specific subject into a pre-trained LMM via a learnable prompt composed of an identifier token and latent tokens, freezing most model weights to avoid forgetting. It uses hard negative mining to capture fine-grained visual differences and trains on recognition and QA data, including text-only QA to reduce reliance on test-time images. The approach delivers state-of-the-art recognition and QA performance with a small set of trainable tokens (k=16, ~5 training images per subject) and shows clear gains over prompting baselines and MyVLM without external recognizers. Overall, the work demonstrates efficient, integrated personalization for individual subjects and points toward leveraging user metadata for richer real-world deployments.

Abstract

Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog). Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for my dog's birthday?"; as opposed to a generic inquiry about "What should I buy for a dog's birthday?". Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a man is holding a cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).

Yo'LLaVA: Your Personalized Language and Vision Assistant

TL;DR

Yo'LLaVA tackles personalized multimodal conversation by embedding a user-specific subject into a pre-trained LMM via a learnable prompt composed of an identifier token and latent tokens, freezing most model weights to avoid forgetting. It uses hard negative mining to capture fine-grained visual differences and trains on recognition and QA data, including text-only QA to reduce reliance on test-time images. The approach delivers state-of-the-art recognition and QA performance with a small set of trainable tokens (k=16, ~5 training images per subject) and shows clear gains over prompting baselines and MyVLM without external recognizers. Overall, the work demonstrates efficient, integrated personalization for individual subjects and points toward leveraging user metadata for richer real-world deployments.

Abstract

Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog). Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for my dog's birthday?"; as opposed to a generic inquiry about "What should I buy for a dog's birthday?". Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a man is holding a cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).
Paper Structure (28 sections, 1 equation, 5 figures, 28 tables)

This paper contains 28 sections, 1 equation, 5 figures, 28 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Yo'LLaVA: Personalizing LMMs to Understand User-Specific Concepts
  4. Personalizing the Subject as a Learnable Prompt
  5. Enhancing Recognition with Hard Negative Mining
  6. Learning to Engage in Natural Conversations about the Subject
  7. Experimental Setup
  8. Results
  9. Recognition Ability
  10. Question and Answering
  11. Comparison with MyVLM
  12. Ablation Studies
  13. Conclusion
  14. Acknowledgements.
  15. Broader Impact
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Given just a few images of a novel subject (e.g., a dog named <bo>), Yo'LLaVA learns to facilitate textual/visual conversations centered around that subject.
  • Figure 2: Training pipeline.
  • Figure 3: Training data creation.
  • Figure 4: Number of Prompting Tokens vs. Recognition Ability.
  • Figure 5: Ablation studies on the number of trainable tokens and training images. Overall, the model's ability to recognize personalized subjects increases as these parameters increase.