Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Multimodal Entity Linking

Senbao Shi, Zhenran Xu, Baotian Hu, Min Zhang

TL;DR

GEMEL is a Generative Multimodal Entity Linking framework based on LLMs, which directly generates target entity names and is compatible with any off-the-shelf language model, paving the way towards an efficient and general solution for utilizing LLMs in the MEL task.

Abstract

Multimodal Entity Linking (MEL) is the task of mapping mentions with multimodal contexts to the referent entities from a knowledge base. Existing MEL methods mainly focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters, which can be prohibitively costly and difficult to scale in the era of Large Language Models (LLMs). In this work, we propose GEMEL, a Generative Multimodal Entity Linking framework based on LLMs, which directly generates target entity names. We keep the vision and language model frozen and only train a feature mapper to enable cross-modality interactions. To adapt LLMs to the MEL task, we leverage the in-context learning capability of LLMs by retrieving multimodal instances as demonstrations. Extensive experiments show that, with only ~0.3% of the model parameters fine-tuned, GEMEL achieves state-of-the-art results on two well-established MEL datasets (7.7% accuracy gains on WikiDiverse and 8.8% accuracy gains on WikiMEL). The performance gain stems from mitigating the popularity bias of LLM predictions and disambiguating less common entities effectively. Further analysis verifies the generality and scalability of GEMEL. Our framework is compatible with any off-the-shelf language model, paving the way towards an efficient and general solution for utilizing LLMs in the MEL task. Our code is available at https://github.com/HITsz-TMG/GEMEL.

Generative Multimodal Entity Linking

TL;DR

GEMEL is a Generative Multimodal Entity Linking framework based on LLMs, which directly generates target entity names and is compatible with any off-the-shelf language model, paving the way towards an efficient and general solution for utilizing LLMs in the MEL task.

Abstract

Multimodal Entity Linking (MEL) is the task of mapping mentions with multimodal contexts to the referent entities from a knowledge base. Existing MEL methods mainly focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters, which can be prohibitively costly and difficult to scale in the era of Large Language Models (LLMs). In this work, we propose GEMEL, a Generative Multimodal Entity Linking framework based on LLMs, which directly generates target entity names. We keep the vision and language model frozen and only train a feature mapper to enable cross-modality interactions. To adapt LLMs to the MEL task, we leverage the in-context learning capability of LLMs by retrieving multimodal instances as demonstrations. Extensive experiments show that, with only ~0.3% of the model parameters fine-tuned, GEMEL achieves state-of-the-art results on two well-established MEL datasets (7.7% accuracy gains on WikiDiverse and 8.8% accuracy gains on WikiMEL). The performance gain stems from mitigating the popularity bias of LLM predictions and disambiguating less common entities effectively. Further analysis verifies the generality and scalability of GEMEL. Our framework is compatible with any off-the-shelf language model, paving the way towards an efficient and general solution for utilizing LLMs in the MEL task. Our code is available at https://github.com/HITsz-TMG/GEMEL.
Paper Structure (18 sections, 2 equations, 4 figures, 7 tables)

This paper contains 18 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Textual EL
  4. Multimodal EL
  5. LLMs for Vision-language Tasks
  6. Methodology
  7. Problem Formulation
  8. Feature Alignment
  9. Language Model Generation
  10. Experiments
  11. Experimental Setup
  12. Main Results
  13. Analysis
  14. Generality and Scalability
  15. Demonstration Selection
  16. ...and 3 more sections

Figures (4)

  • Figure 1: An example of multimodal entity linking with mention underlined in the text. Based solely on text, it is hard to determine whether “Harry Potter” should be linked to the films or the novels. With image as context, "Harry Potter" can be easily linked to the Harry Potter film series.
  • Figure 2: Overview of the GEMEL method. Given the multimodal mention context, GEMEL first uses a feature mapper to transform image features to visual prefix in the textual space. Then, GEMEL leverages the capabilities of LLM to directly generate the target entity name (e.g. "wheelchair fencing"), with $n$ retrieved multimodal instances as in-context demonstrations. GEMEL applies a constrained decoding strategy to efficiently search the valid entity space. The mention in the text is underlined.
  • Figure 3: Results of scaling up parameters of OPT. As language models continue to scale up, GEMEL consistently exhibits enhanced performance in the MEL task.
  • Figure 4: Case study. For underlined mentions, green and red text mean successful and failed predictions, respectively. The text-only prediction results are obtained by GEMEL without utilizing visual information.