Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning

TL;DR

Open-domain visual entity recognition requires linking images to a vast and evolving set of real-world concepts, including unseen entities. The authors propose KnowCoL, a knowledge-guided contrastive learning framework that jointly embeds images, text, and Wikidata/Wikipedia knowledge into a shared latent space to enable zero-shot recognition and disambiguation. On the OVEN benchmark, KnowCoL variants outperform dual-encoder and generative baselines, with KnowCoL-bigG achieving a high harmonic mean while using far fewer parameters than large generative models. This work highlights the value of integrating structured knowledge with multi-modal priors to scale open-domain recognition and points to future directions in non-Euclidean embedding spaces for hierarchical knowledge modeling.

Abstract

Open-domain visual entity recognition aims to identify and link entities depicted in images to a vast and evolving set of real-world concepts, such as those found in Wikidata. Unlike conventional classification tasks with fixed label sets, it operates under open-set conditions, where most target entities are unseen during training and exhibit long-tail distributions. This makes the task inherently challenging due to limited supervision, high visual ambiguity, and the need for semantic disambiguation. We propose a Knowledge-guided Contrastive Learning (KnowCoL) framework that combines both images and text descriptions into a shared semantic space grounded by structured information from Wikidata. By abstracting visual and textual inputs to a conceptual level, the model leverages entity descriptions, type hierarchies, and relational context to support zero-shot entity recognition. We evaluate our approach on the OVEN benchmark, a large-scale open-domain visual recognition dataset with Wikidata IDs as the label space. Our experiments show that using visual, textual, and structured knowledge greatly improves accuracy, especially for rare and unseen entities. Our smallest model improves the accuracy on unseen entities by 10.5% compared to the state-of-the-art, despite being 35 times smaller.

Figures (4)

• Figure 1: (a) The open-domain visual entity recognition task aims to link an image and a text query to a corresponding Wikidata entity. The text query expresses the intent to the image. (b) Both visual and textual inputs can be mapped to a shared semantic space enriched with structured knowledge, such as entity relations (e.g., instance-of, part-of, subclass-of) from Wikidata. Leveraging this additional knowledge source enables contextual reasoning and effective disambiguation of entities.
• Figure 2: Framework Overview. Given an input image and text query (left), CLIP encoders extract image and text features. LP$_1$ and LP$_2$ present linear project layers for fine-tuning. A fuser module combines these to represent the intended entity. On the right, prior knowledge of the entity (Q160659 – London Eye)—including sample images, textual descriptions, and knowledge graph structure—is embedded using CLIP’s image and text encoders, as well as knowledge graph embedding (KGE) method. A proxy loss function is employed to align the semantic space (KGEs), visual space (image embeddings), and textual space (text embeddings). Additionally, a knowledge embedding (KE) loss is applied to learn KGEs from the structured knowledge. Finally, an alignment loss is used to align the different input representation with the corresponding entity embedding.
• Figure 3: Node embeddings act as proxies, aligned with their corresponding image and text embeddings. Knowledge (from the KG) is captured through relational embeddings, connecting node embeddings in the latent space. Q160659, Q446, P31 represent London Eye, Wheel, and Instance of.
• Figure 4: Ablation Studies for hyperparameter $\beta_1$, $\beta_2$, latent space dimension $d_e$, and temperature $\tau$. HM indicates the harmonic mean of accuracies of seen and unseen entities. - represents the default setting of KnowCoL approach.