Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities

TL;DR

The paper introduces Oven as a benchmark for open-domain visual entity recognition, grounding millions of Wikipedia entities to a universal label space via image-text queries. It constructs Oven-Wiki by unifying 14 datasets and annotating a gold evaluation set, enabling evaluation over 6 million candidate entities. Through systematic fine-tuning of PaLI-based encoder-decoder and CLIP-based dual encoders, it shows PaLI models achieve strongest overall performance, particularly on unseen entities, while CLIP-based methods excel on tail entities and retrieval aspects. The work highlights the feasibility and challenges of universal visual grounding, analyzes model behavior and error modes, and suggests directions for scalable, knowledge-infused vision-language systems.

Abstract

Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit strong generalization on various visual domains and tasks. However, existing image classification benchmarks often evaluate recognition on a specific domain (e.g., outdoor images) or a specific task (e.g., classifying plant species), which falls short of evaluating whether pre-trained foundational models are universal visual recognizers. To address this, we formally present the task of Open-domain Visual Entity recognitioN (OVEN), where a model need to link an image onto a Wikipedia entity with respect to a text query. We construct OVEN-Wiki by re-purposing 14 existing datasets with all labels grounded onto one single label space: Wikipedia entities. OVEN challenges models to select among six million possible Wikipedia entities, making it a general visual recognition benchmark with the largest number of labels. Our study on state-of-the-art pre-trained models reveals large headroom in generalizing to the massive-scale label space. We show that a PaLI-based auto-regressive visual recognition model performs surprisingly well, even on Wikipedia entities that have never been seen during fine-tuning. We also find existing pretrained models yield different strengths: while PaLI-based models obtain higher overall performance, CLIP-based models are better at recognizing tail entities.

Figures (9)

• Figure 1: An illustration of the proposed Oven task. Examples on the right are sampled from the constructed Oven-Wiki dataset. Oven aims at recognizing entities physically presented in the image or can be directly inferred from the image.
• Figure 2: Illustration on two Oven Models.
• Figure 3: Dataset Statistics of the Oven-Wiki. Left: Distribution of super-categories of entities that have positive examples (See Appendix for more details). Mid: Statistics of different splits of the Oven-Wiki. Right: Properties of the Wikipedia dump-2022/10/01.
• Figure 4: Fine-tuning PaLI or CLIP2CLIP for large # of steps increases the seen entity accuracy but hurts the unseen entity accuracy.
• Figure 5: Impact of # Wikipedia Candidates on PaLI and CLIP2CLIP. Increasing the size of Wikipedia makes the tasks difficult.