Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

Shezheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, Meng Wang

TL;DR

This work tackles Multimodal Entity Linking by reframing MEL as neural text matching and introducing a Dual-Way Enhanced (DWE) framework. DWE refines the query through cross-modal enhancers and three specialized units (textual, attribute, and vision) to fuse multimodal signals, while enriching entity semantics with Wikipedia descriptions to reduce mismatch with KG representations. The approach employs gated fusion and a composite loss that combines triplet and cross-modal alignment terms, and it demonstrates state-of-the-art results across Richpedia, WikiMEL, and Wikidiverse. The findings highlight the value of fine-grained visual attributes and cross-modal alignment for robust MEL and offer a practical, publicly available implementation.

Abstract

Multimodal Entity Linking (MEL) aims at linking ambiguous mentions with multimodal information to entity in Knowledge Graph (KG) such as Wikipedia, which plays a key role in many applications. However, existing methods suffer from shortcomings, including modality impurity such as noise in raw image and ambiguous textual entity representation, which puts obstacles to MEL. We formulate multimodal entity linking as a neural text matching problem where each multimodal information (text and image) is treated as a query, and the model learns the mapping from each query to the relevant entity from candidate entities. This paper introduces a dual-way enhanced (DWE) framework for MEL: (1) our model refines queries with multimodal data and addresses semantic gaps using cross-modal enhancers between text and image information. Besides, DWE innovatively leverages fine-grained image attributes, including facial characteristic and scene feature, to enhance and refine visual features. (2)By using Wikipedia descriptions, DWE enriches entity semantics and obtains more comprehensive textual representation, which reduces between textual representation and the entities in KG. Extensive experiments on three public benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance, indicating the superiority of our model. The code is released on https://github.com/season1blue/DWE

A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

TL;DR

This work tackles Multimodal Entity Linking by reframing MEL as neural text matching and introducing a Dual-Way Enhanced (DWE) framework. DWE refines the query through cross-modal enhancers and three specialized units (textual, attribute, and vision) to fuse multimodal signals, while enriching entity semantics with Wikipedia descriptions to reduce mismatch with KG representations. The approach employs gated fusion and a composite loss that combines triplet and cross-modal alignment terms, and it demonstrates state-of-the-art results across Richpedia, WikiMEL, and Wikidiverse. The findings highlight the value of fine-grained visual attributes and cross-modal alignment for robust MEL and offer a practical, publicly available implementation.

Abstract

Multimodal Entity Linking (MEL) aims at linking ambiguous mentions with multimodal information to entity in Knowledge Graph (KG) such as Wikipedia, which plays a key role in many applications. However, existing methods suffer from shortcomings, including modality impurity such as noise in raw image and ambiguous textual entity representation, which puts obstacles to MEL. We formulate multimodal entity linking as a neural text matching problem where each multimodal information (text and image) is treated as a query, and the model learns the mapping from each query to the relevant entity from candidate entities. This paper introduces a dual-way enhanced (DWE) framework for MEL: (1) our model refines queries with multimodal data and addresses semantic gaps using cross-modal enhancers between text and image information. Besides, DWE innovatively leverages fine-grained image attributes, including facial characteristic and scene feature, to enhance and refine visual features. (2)By using Wikipedia descriptions, DWE enriches entity semantics and obtains more comprehensive textual representation, which reduces between textual representation and the entities in KG. Extensive experiments on three public benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance, indicating the superiority of our model. The code is released on https://github.com/season1blue/DWE
Paper Structure (24 sections, 13 equations, 2 figures, 6 tables)

This paper contains 24 sections, 13 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Notation and Task Definition
  5. Preliminary
  6. Multimodal Information Enhanced Query
  7. Cross-modal enhancer.
  8. Text-enhanced unit.
  9. Attribute-enhanced unit.
  10. Vision-enhanced unit.
  11. Gated Fusion and Training Loss
  12. Cross-modal alignment.
  13. Experiments
  14. Datasets and Settings
  15. Metrics
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Example of entity linking for the mention Trump.
  • Figure 2: Overview of DWE. Input consists of image $I$, text $t$, and mention $m$. Object detection is applied to extract object feature $d_i$ from image. Facial feature $f$ and scene feature $s$ (i.e. Adjective-Noun Pairs(ANPs)) is retrieved from image. The fusion feature $g$ is combination of image-enhanced feature $m_v$, text-enhanced feature $m_t$ and attribute-enhanced feature $m_s$.