Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrieval

TL;DR

Jarvis tackles the challenge of personalizing vision-language models by externalizing user-specific evidence into reusable KV caches that ground answers without updating model parameters. It builds offline text metadata and discriminative hard patches, materializes them as external KV states, and retrieves a compact evidence prefix at inference to enable single-pass decoding with a frozen backbone. The approach achieves state-of-the-art results on text-only QA and visual QA across Yo'LLaVA and MC-LLaVA, while significantly reducing latency and improving throughput through prefilled caches and short prompts. The work is augmented with a patch-centric, fine-grained dataset to stress attribute-level grounding and distractor robustness, and it points to practical paths for deploying personalized AI assistants with scalable memory and privacy considerations.

Abstract

The rapid development of Vision-language models (VLMs) enables open-ended perception and reasoning. Recent works have started to investigate how to adapt general-purpose VLMs into personalized assistants. Even commercial models such as ChatGPT now support model personalization by incorporating user-specific information. However, existing methods either learn a set of concept tokens or train a VLM to utilize user-specific information. However, both pipelines struggle to generate accurate answers as personalized assistants. We introduce Jarvis, an innovative framework for a personalized AI assistant through personal KV-Cache retrieval, which stores user-specific information in the KV-Caches of both textual and visual tokens. The textual tokens are created by summarizing user information into metadata, while the visual tokens are produced by extracting distinct image patches from the user's images. When answering a question, Jarvis first retrieves related KV-Caches from personal storage and uses them to ensure accuracy in responses. We also introduce a fine-grained benchmark built with the same distinct image patch mining pipeline, emphasizing accurate question answering based on fine-grained user-specific information. Jarvis is capable of providing more accurate responses, particularly when they depend on specific local details. Jarvis achieves state-of-the-art results in both visual question answering and text-only tasks across multiple datasets, indicating a practical path toward personalized AI assistants. The code and dataset will be released.

Figures (4)

• Figure 1: Qualitative comparison with Yo’LLaVA and RAP-LLaVA across three personalized scenarios. (1) Detailed object recall:Jarvis correctly recovers fine-grained, instance-specific details (e.g., the small peach ornament) where baselines hallucinate. (2) Contextualized personalized description:Jarvis follows the instruction to contrast with the user’s usual appearance and produces precise, non-generic attributes, while baselines either ignore the comparison or misdescribe. (3) Personalized property inference/reasoning: From visual cues, Jarvis infers abstract cultural aesthetics and functional features (e.g., “kawaii” motifs) with higher faithfulness and completeness.
• Figure 2: Model overview. (a) Offline evidence construction: text metadata synthesis and concept-only hard patch mining. (b) Query-time retrieval: similarity search over text and visual evidence. (c) KV prefill & single-pass decoding: precompute concept KV states and reuse them during decoding.
• Figure 3: Concept-only hard patch mining. We localize the concept, fuse difficulty and text–relevance cues inside the mask, then grid-score to extract top-$k$ informative crops, which populate the visual-indexed Hard Patch Pool used by the retriever.
• Figure 4: Throughput and latency under personalization. (Left) QPS vs. $Q$. (Right) average per-turn latency vs. $Q$. All methods share hardware/decoding, meaning that 95% CIs are consistent across trials. Higher is better for QPS; lower is better for latency.