Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation

Cong-Duy Nguyen, Xiaobao Wu, Thong Nguyen, Shuai Zhao, Khoi Le, Viet-Anh Nguyen, Feng Yichao, Anh Tuan Luu

TL;DR

This paper tackles multimodal entity linking (MEL) by addressing two key weaknesses of prior contrastive-learning-based MEL: easy negative samples and misalignment due to visual variability. It introduces JD-CCL, a Jaccard-distance-based conditional sampling strategy that selects hard negatives from meta-attribute-similar entities, and CVaCPT, a visual augmentation module that uses diffusion-generated synthetic images and contextual cues to controllably adjust patch representations. The approach is evaluated on WikiDiverse, RichpediaMEL, and WikiMEL, where it achieves state-of-the-art performance and notable gains in top-1 accuracy and overall ranking metrics, outperforming strong baselines such as CLIP, ViLT, ALBEF, METER, and MIMIC. Overall, the combination of attribute-aware negative sampling and context-driven, multi-view visual augmentation improves MEL robustness, especially in large knowledge bases with diverse visual content, and offers a practical path toward more reliable multimodal disambiguation.

Abstract

Previous research on multimodal entity linking (MEL) has primarily employed contrastive learning as the primary objective. However, using the rest of the batch as negative samples without careful consideration, these studies risk leveraging easy features and potentially overlook essential details that make entities unique. In this work, we propose JD-CCL (Jaccard Distance-based Conditional Contrastive Learning), a novel approach designed to enhance the ability to match multimodal entity linking models. JD-CCL leverages meta-information to select negative samples with similar attributes, making the linking task more challenging and robust. Additionally, to address the limitations caused by the variations within the visual modality among mentions and entities, we introduce a novel method, CVaCPT (Contextual Visual-aid Controllable Patch Transform). It enhances visual representations by incorporating multi-view synthetic images and contextual textual representations to scale and shift patch representations. Experimental results on benchmark MEL datasets demonstrate the strong effectiveness of our approach.

Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation

TL;DR

This paper tackles multimodal entity linking (MEL) by addressing two key weaknesses of prior contrastive-learning-based MEL: easy negative samples and misalignment due to visual variability. It introduces JD-CCL, a Jaccard-distance-based conditional sampling strategy that selects hard negatives from meta-attribute-similar entities, and CVaCPT, a visual augmentation module that uses diffusion-generated synthetic images and contextual cues to controllably adjust patch representations. The approach is evaluated on WikiDiverse, RichpediaMEL, and WikiMEL, where it achieves state-of-the-art performance and notable gains in top-1 accuracy and overall ranking metrics, outperforming strong baselines such as CLIP, ViLT, ALBEF, METER, and MIMIC. Overall, the combination of attribute-aware negative sampling and context-driven, multi-view visual augmentation improves MEL robustness, especially in large knowledge bases with diverse visual content, and offers a practical path toward more reliable multimodal disambiguation.

Abstract

Previous research on multimodal entity linking (MEL) has primarily employed contrastive learning as the primary objective. However, using the rest of the batch as negative samples without careful consideration, these studies risk leveraging easy features and potentially overlook essential details that make entities unique. In this work, we propose JD-CCL (Jaccard Distance-based Conditional Contrastive Learning), a novel approach designed to enhance the ability to match multimodal entity linking models. JD-CCL leverages meta-information to select negative samples with similar attributes, making the linking task more challenging and robust. Additionally, to address the limitations caused by the variations within the visual modality among mentions and entities, we introduce a novel method, CVaCPT (Contextual Visual-aid Controllable Patch Transform). It enhances visual representations by incorporating multi-view synthetic images and contextual textual representations to scale and shift patch representations. Experimental results on benchmark MEL datasets demonstrate the strong effectiveness of our approach.
Paper Structure (38 sections, 7 equations, 6 figures, 7 tables)

This paper contains 38 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Entity Linking
  4. Contrastive learning
  5. Problem Formulation
  6. Methodology
  7. Overall Architecture
  8. Visual Encoding
  9. Textual Encoding
  10. Matcher
  11. Objective
  12. Conditional Contrastive learning
  13. Conditional sampling
  14. Jaccard Distance-based CCL
  15. Contextual Visual-aid Controllable Patch Transformation
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Example from WikiDiverse: The first two samples are the sentence input and the ground truth entity for its mention. The last two are other "Supreme Court" entities taken from the knowledge base.
  • Figure 2: Illustration of challenge involving disparities within the visual modality. A sample from WikiDiverse includes the mention sentence and its image talking about the actor Doohan. Input sentence contains multiple entities from the knowledge base: Doohan, Star Trek, Scotty, Wikipedia.
  • Figure 3: Task definition of Multimodal Entity Linking.
  • Figure 4: The overall architecture of MEL problem.
  • Figure 5: The overall architecture of Contextual Visual-aid Controllable Patch Transformation. The model incorporates a text-to-image diffuser, visual encoder, and text encoder, followed by a contextual visual-aid controllable patch transform (CVaCPT). The generated images and input sentence will help the model selectively control visual representation through the CVaCPT, including Contextual encoder, Max pooling, and Controllable patch transform.
  • ...and 1 more figures