OLIVE: Object Level In-Context Visual Embeddings

TL;DR

OLIVE tackles the bottleneck of fine-grained object grounding in vision-language models by introducing a lightweight object encoder that encodes objects into a single vector aligned with the language model's space, thus avoiding exhaustive patch-level fusion. It couples this with a region-level retrieval system and code-switch prompts to enable in-context object reasoning across multiple images, enabling rapid adaptation to unseen concepts without retraining. The approach achieves competitive object classification and captioning while demonstrating zero-shot generalization, robustness in challenging contexts, and efficient training and inference. This combination of compact object representations and retrieval-augmented prompting holds practical potential for domain-specific tasks where labeled data is scarce and unseen objects frequently appear.

Abstract

Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with fine-grained object-level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object-level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitating rapid adaptation to new objects without additional training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance, while also offering zero-shot generalization and robustness to visually challenging contexts.

Figures (13)

• Figure 1: An overview of our method which consists of three main components, described in detail in Section \ref{['sec:method']}. The object encoder (green) is the only module with required trainable parameters. Note: the prompt in the LLM decoder is modified slightly for visual clarity. The exact prompt can be found in Appendix \ref{['sec:prompts']}.
• Figure 2: Examples showcasing the benefit of using retrieval for out of distribution objects. Despite not being trained with any images of sharks or turtles, $\textsc{OLIVE-RG}$ can describe them zero shot by adding a few pictures of them in the retrieval set.
• Figure 3: Classification accuracy on the most uncommon objects in the COCO dataset. We use a Llama 2 decoder backbone and numbers below the object classes indicate their proportion of the retrieval set. Combining retrieval with generative methods improves performance and increasing vision encoder resolution provides further gains.
• Figure 4: A challenging visual example in which the background of the image does not correspond with the query object. Methods which cross-attend to the whole image struggle to identify the snowboard, while our object representation enables more accurate description.
• Figure 5: Context length of different VLMs when prompted with multimodal input. Models that represent images with many patch tokens or learned latents incur higher costs with more in-context examples.