Contextualized Visual Personalization in Vision-Language Models

TL;DR

The paper tackles the problem of contextualized visual personalization in vision-language models by treating personalized image captioning as a core proxy task. It proposes CoViP, a unified framework that decouples a shared contextual encoder from a task-specific generator, trains with reinforcement learning to optimize verifiable rewards, and uses caption-augmented generation to boost downstream personalization. A novel personalized image captioning benchmark and CapEval-QAs evaluation protocol quantify grounding and retrieval of user-specific context, while a diagnostic suite (LSD, LAR, ITR) assesses robustness beyond surface-level cues. Empirical results show that CoViP consistently improves caption grounding and downstream personalization across open-source and proprietary VLMs, with captioning-focused pretraining providing robust, generalizable gains. The work highlights the importance of grounding and retrieval over mere recognition and discusses limitations of synthetic data and future directions toward real-world, privacy-preserving personalization.

Abstract

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

Figures (14)

• Figure 1: Qualitative example of the use-case for contextual visual personalization in VLMs. Note that our CoViP effectively responds to the question while integrating the mentioned personal details from the given multimodal contexts.
• Figure 2: Illustration of the proposed personalized image captioning benchmark construction.
• Figure 3: Visualization of diagnostic personalization tasks.
• Figure 4: Results of the human preference evaluation. Here, Win denotes the win rates of CoViP compared to the baseline.
• Figure 5: Visualization of comparative results for visual-triggered personalization diagnostics. Scores indicate relative performance.