ObjEmbed: Towards Universal Multimodal Object Embeddings

TL;DR

ObjEmbed tackles the challenge of fine-grained object–text alignment and localization by introducing object-centric dual embeddings (object and IoU) within a single-tower MLLM framework. It processes all RoIs in a single forward pass, jointly learning region-level and image-level representations through three losses and a unified objective $\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{region} + \lambda_2 \mathcal{L}_{image} + \lambda_3 \mathcal{L}_{iou}$. The approach yields strong results across 18 benchmarks, excelling in local retrieval and achieving competitive global retrieval while maintaining robust object-level discrimination. This work provides a versatile, efficient baseline for universal object embeddings with clear potential for improvement via better proposals and larger-scale data.

Abstract

Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in vision-language understanding. While recent multimodal embedding models excel at global image-text alignment, they often struggle with fine-grained alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM embedding model that decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image retrieval, and global image retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary embeddings for each region: an object embedding for semantic matching and an IoU embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse benchmarks demonstrates its strong semantic discrimination.

Figures (5)

• Figure 1: ObjEmbed achieves balanced and superior performance across a wide span of benchmarks.
• Figure 2: The architecture of ObjEmbed. ObjEmbed is a single-tower model built upon a large multimodal language model, enhanced with an object projector and five special tokens ($\langle$object$\rangle$, $\langle$iou$\rangle$, $\langle$global$\rangle$, $\langle$local_text$\rangle$, and $\langle$global_text$\rangle$) whose hidden states from the last layer are used as embeddings. ObjEmbed encodes all object embeddings, IoU embeddings, and the global image embeddings in a single forward pass. The visual embeddings and the text embeddings share the same LLM encoder. For detected objects, object embeddings are initialized from RoI features. And the final matching score is computed as the product of the predicted IoU score (predicted from IoU embeddings) and the classification score (predicted from object embeddings and local text embeddings).
• Figure 3: Visualizations of referring expression comprehension results with text queries and image queries.
• Figure 4: Visualizations of retrieval results on SORCE-1K. Our ObjEmbed successfully ranks the target image as the top result and accurately localizes the target objects (highlighted with red bounding boxes). In contrast, global image embedding models, like Qwen3-VL-Embedding 8B, tend to overlook small objects.
• Figure 5: Visualizations of self-annotated data. Each image is annotated with high-quality image-level and object-level captions. Images come from SA-1B kirillov2023segment.