IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models

TL;DR

IIR-VLM addresses the gap in instance-level recognition for large vision-language models by augmenting a base VLM with pre-trained ILR expert encoders and a lightweight two-stage in-context training regime. The method fuses identity-discriminative features into the VLM's visual tokens via attention-based fusion, and trains Stage 1 for in-context instance matching followed by Stage 2 for instance-aware captioning, enabling one-shot learning of new instances without per-instance fine-tuning. A new cross-category ILR benchmark demonstrates state-of-the-art matching and captioning performance, with advantages that grow as tasks become more visually challenging. The findings underscore the value of category-specific ILR knowledge for fine-grained discrimination and hold promise for practical personalization in settings such as smart-home visual assistants.

Abstract

Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical application of VLMs, e.g. where recognizing familiar people and objects is crucial for effective visual understanding. Existing solutions typically learn to recognize instances one at a time using instance-specific datasets, which not only incur substantial data collection and training costs but also struggle with fine-grained discrimination. In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances, which enables VLMs to learn new instances in-context in a one-shot manner. Further, IIR-VLM leverages this knowledge for instance-aware visual understanding. We validate IIR-VLM's efficacy on existing instance personalization benchmarks. Finally, we demonstrate its superior ILR performance on a challenging new benchmark, which assesses ILR capabilities across varying difficulty and diverse categories, with person, face, pet and general objects as the instances at task.

Figures (6)

• Figure 1: Pipeline of our proposed IIR-VLM. IIR-VLM learns instances in-context from the gallery and performs visual understanding tasks related to the query. All images are processed by both the original general-purpose visual encoder and an instance-level recognition expert encoder, which is trained exclusively on instance classification tasks. Expert features are re-weighted by an attention score $A$ to indicate instance presence in each original feature token. Re-weighted features are added to the original tokens to provide discriminative features beneficial for ILR tasks.
• Figure 2: Samples of our instance-level recognition benchmark and corresponding model outputs. We cover a wide span of categories, including household items (object), persons, faces, and pets. In each conversation, models take a group of images as input, where the leftmost image represents the query image at task, and five visually similar gallery images follow. Models are required to select the gallery image with the matching instance (in green), and generate captions of the query image referring to the correct instance.
• Figure 3: Comparison between models with and without ILR expert encoder across different difficulty levels. Numbers on the right indicate the relative accuracy improvements.
• Figure 4: Additional examples of our instance-level recognition dataset along with corresponding model outputs. IIR-VLM is consistently capable of correctly identifying the matching item in the gallery (green frame) and referring to the item in caption generation.
• Figure 5: Visualizing the effect of the similarity threshold $\tau$, which controls the visual similarity between gallery images and the query image. As the threshold increases, the gallery is populated with candidates more similar to the query, thereby making the benchmark more challenging.