Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VP-MEL: Visual Prompts Guided Multimodal Entity Linking

Hongze Mi, Jinyuan Li, Xuying Zhang, Haoran Cheng, Jiahao Wang, Di Sun, Gang Pan

TL;DR

This work introduces Visual Prompts guided Multimodal Entity Linking (VP-MEL) to link image regions to KB entities without relying on textual mention words. It presents VPWiki, a VP-MEL dataset, and the IIER framework, which uses visual prompts to enhance local image features and Detective-VLM to generate supplementary textual cues, combined via three multimodal interaction units and contrastive learning. IIER achieves state-of-the-art performance on VP-MEL and strong results on MEL, demonstrating the value of visual prompts and implicit knowledge in multimodal entity linking. The approach broadens MEL applicability to real-world scenarios where textual cues are scarce or ambiguous, enabling more robust and flexible linking across modalities.

Abstract

Multimodal entity linking (MEL), a task aimed at linking mentions within multimodal contexts to their corresponding entities in a knowledge base (KB), has attracted much attention due to its wide applications in recent years. However, existing MEL methods often rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. This reliance causes MEL to struggle with accurately retrieving entities in certain scenarios, especially when the focus is on image objects or mention words are missing from the text. To solve these issues, we introduce a Visual Prompts guided Multimodal Entity Linking (VP-MEL) task. Given a text-image pair, VP-MEL aims to link a marked region (i.e., visual prompt) in an image to its corresponding entities in the knowledge base. To facilitate this task, we present a new dataset, VPWiki, specifically designed for VP-MEL. Furthermore, we propose a framework named IIER, which enhances visual feature extraction using visual prompts and leverages the pretrained Detective-VLM model to capture latent information. Experimental results on the VPWiki dataset demonstrate that IIER outperforms baseline methods across multiple benchmarks for the VP-MEL task.

VP-MEL: Visual Prompts Guided Multimodal Entity Linking

TL;DR

This work introduces Visual Prompts guided Multimodal Entity Linking (VP-MEL) to link image regions to KB entities without relying on textual mention words. It presents VPWiki, a VP-MEL dataset, and the IIER framework, which uses visual prompts to enhance local image features and Detective-VLM to generate supplementary textual cues, combined via three multimodal interaction units and contrastive learning. IIER achieves state-of-the-art performance on VP-MEL and strong results on MEL, demonstrating the value of visual prompts and implicit knowledge in multimodal entity linking. The approach broadens MEL applicability to real-world scenarios where textual cues are scarce or ambiguous, enabling more robust and flexible linking across modalities.

Abstract

Multimodal entity linking (MEL), a task aimed at linking mentions within multimodal contexts to their corresponding entities in a knowledge base (KB), has attracted much attention due to its wide applications in recent years. However, existing MEL methods often rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. This reliance causes MEL to struggle with accurately retrieving entities in certain scenarios, especially when the focus is on image objects or mention words are missing from the text. To solve these issues, we introduce a Visual Prompts guided Multimodal Entity Linking (VP-MEL) task. Given a text-image pair, VP-MEL aims to link a marked region (i.e., visual prompt) in an image to its corresponding entities in the knowledge base. To facilitate this task, we present a new dataset, VPWiki, specifically designed for VP-MEL. Furthermore, we propose a framework named IIER, which enhances visual feature extraction using visual prompts and leverages the pretrained Detective-VLM model to capture latent information. Experimental results on the VPWiki dataset demonstrate that IIER outperforms baseline methods across multiple benchmarks for the VP-MEL task.

Paper Structure

This paper contains 44 sections, 37 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Entity Linking
  4. Vision Prompt
  5. Dataset
  6. Data Collection.
  7. Annotation Design.
  8. Annotation Procedure.
  9. Dataset Analysis.
  10. Task Formulation
  11. Methodology
  12. Visual Encoder
  13. Detective-VLM
  14. Textual Encoder
  15. Multimodal Feature Interaction
  16. ...and 29 more sections

Figures (7)

  • Figure 1: Comparison between MEL and VP-MEL tasks. MEL is typically limited to selecting mentions from text. In contrast, VP-MEL addresses this limitation by using visual prompts to link specific regions in the images to the correct entities in the knowledge base.
  • Figure 2: An example from VPWiki. GT denotes the ground truth entity. The red box in the left image represents the visual prompt annotated for the VP-MEL task.
  • Figure 3: More statistics of VPWiki. (a) Distribution of entity types. (b) Distribution of the number of candidate entities per mention.
  • Figure 4: The overall architecture of Implicit Information-Enhanced Reasoning (IIER) framework. The image-text pairs of the Mention and Entity are used together as input. Specifically, Mention Text is the sentence corresponding to Mention Image, while Entity Text consists of Entity Name and Entity Attribute corresponding to the Entity Image in the Knowledge Base.
  • Figure 5: Entity type distribution of WIKIDiverse.
  • ...and 2 more figures