TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant

TL;DR

The paper tackles the challenge of enabling MLLMs to engage in long-context personalization by introducing LCMP, a benchmark designed to evaluate how well models track and utilize both short-term and long-term attributes of personalized concepts across multi-turn vision-language dialogues. It proposes TAME, a training-free, state-aware assistant built on a double-memory architecture (dynamic state memory and static personalized memory) and a Retrieve-then-Align Augmented Generation (RA2G) paradigm to ground responses in the most relevant memories. LCMP enables automatic evaluation of context-aware personalized responses, and comprehensive experiments show that TAME achieves state-of-the-art performance on LCMP by effectively managing memory and aligning retrieved context with user Queries. The work advances practical personal AI by demonstrating how training-free mechanisms can handle evolving user-specific concepts in extended interactions, with potential impact on real-world personalized assistants in multimodal settings.

Abstract

Multimodal Large Language Model (MLLM) Personalization is a critical research problem that facilitates personalized dialogues with MLLMs targeting specific entities (known as personalized concepts). However, existing methods and benchmarks focus on the simple, context-agnostic visual identification and textual replacement of the personalized concept (e.g., "A yellow puppy" -> "Your puppy Mochi"), overlooking the ability to support long-context conversations. An ideal personalized MLLM assistant is capable of engaging in long-context dialogues with humans and continually improving its experience quality by learning from past dialogue histories. To bridge this gap, we propose LCMP, the first Long-Context MLLM Personalization evaluation benchmark. LCMP assesses the capability of MLLMs in perceiving variations of personalized concepts and generating contextually appropriate personalized responses that reflect these variations. As a strong baseline for LCMP, we introduce a novel training-free and state-aware framework TAME. TAME endows MLLMs with double memories to manage the temporal and persistent variations of each personalized concept in a differentiated manner. In addition, TAME incorporates a new training-free Retrieve-then-Align Augmented Generation (RA2G) paradigm. RA2G introduces an alignment step to extract the contextually fitted information from the multi-memory retrieved knowledge to the current questions, enabling better interactions for complex real-world user queries. Experiments on LCMP demonstrate that TAME achieves the best performance, showcasing remarkable and evolving interaction experiences in long-context scenarios.

Figures (25)

• Figure 1: Comparison with prior work: Our envisioned state-aware personalized MLLM assistant learns from historical dialogue to improve the quality of ongoing conversations.
• Figure 2: Conceptual illustration of long-term (persistent) and short-term (temporal) attributes. Short-term attributes can override long-term ones to reflect dynamic context.
• Figure 3: Construction pipeline of LCMP benchmark.
• Figure 4: Evaluation pipeline for LCMP benchmark.
• Figure 5: Overall framework of TAME, a novel training-free and state-aware personalized MLLM assistant.

Theorems & Definitions (6)

• Definition 1: Personalized Concept
• Remark 1
• Definition 2: Short-Term Attribute
• Definition 3: Long-Term Attribute
• Definition 4: Easy VQA Question
• Definition 5: Hard VQA Question