Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MyGram: Modality-aware Graph Transformer with Global Distribution for Multi-modal Entity Alignment

Zhifei Li, Ziyue Qin, Xiangyu Luo, Xiaoju Hou, Yue Zhao, Miao Zhang, Zhifang Huang, Kui Xiao, Bing Yang

TL;DR

MyGram tackles multi-modal entity alignment by introducing a modality-aware diffusion mechanism and a Gram-based global distribution constraint to unify cross-modal representations. It combines modality-specific diffusion, Transformer-based fusion, and a high-dimensional volume regularizer to enforce cross-modal semantic coherence. Empirical results across six dataset configurations show consistent gains over state-of-the-art models, with notable improvements in Hits@1. The work advances robust, structure-aware MMKG alignment and points toward potential integration with large language models for further gains.

Abstract

Multi-modal entity alignment aims to identify equivalent entities between two multi-modal Knowledge graphs by integrating multi-modal data, such as images and text, to enrich the semantic representations of entities. However, existing methods may overlook the structural contextual information within each modality, making them vulnerable to interference from shallow features. To address these challenges, we propose MyGram, a modality-aware graph transformer with global distribution for multi-modal entity alignment. Specifically, we develop a modality diffusion learning module to capture deep structural contextual information within modalities and enable fine-grained multi-modal fusion. In addition, we introduce a Gram Loss that acts as a regularization constraint by minimizing the volume of a 4-dimensional parallelotope formed by multi-modal features, thereby achieving global distribution consistency across modalities. We conduct experiments on five public datasets. Results show that MyGram outperforms baseline models, achieving a maximum improvement of 4.8% in Hits@1 on FBDB15K, 9.9% on FBYG15K, and 4.3% on DBP15K.

MyGram: Modality-aware Graph Transformer with Global Distribution for Multi-modal Entity Alignment

TL;DR

MyGram tackles multi-modal entity alignment by introducing a modality-aware diffusion mechanism and a Gram-based global distribution constraint to unify cross-modal representations. It combines modality-specific diffusion, Transformer-based fusion, and a high-dimensional volume regularizer to enforce cross-modal semantic coherence. Empirical results across six dataset configurations show consistent gains over state-of-the-art models, with notable improvements in Hits@1. The work advances robust, structure-aware MMKG alignment and points toward potential integration with large language models for further gains.

Abstract

Multi-modal entity alignment aims to identify equivalent entities between two multi-modal Knowledge graphs by integrating multi-modal data, such as images and text, to enrich the semantic representations of entities. However, existing methods may overlook the structural contextual information within each modality, making them vulnerable to interference from shallow features. To address these challenges, we propose MyGram, a modality-aware graph transformer with global distribution for multi-modal entity alignment. Specifically, we develop a modality diffusion learning module to capture deep structural contextual information within modalities and enable fine-grained multi-modal fusion. In addition, we introduce a Gram Loss that acts as a regularization constraint by minimizing the volume of a 4-dimensional parallelotope formed by multi-modal features, thereby achieving global distribution consistency across modalities. We conduct experiments on five public datasets. Results show that MyGram outperforms baseline models, achieving a maximum improvement of 4.8% in Hits@1 on FBDB15K, 9.9% on FBYG15K, and 4.3% on DBP15K.
Paper Structure (22 sections, 18 equations, 5 figures, 3 tables)

This paper contains 22 sections, 18 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Uni-modal Entity Alignment
  4. Multi-modal Entity Alignment
  5. Methodology
  6. Preliminaries
  7. Multi-modal Feature Extraction
  8. Modality-aware Diffusion Learning
  9. Multi-modal training and Learning
  10. Experiments
  11. Experimental Settings
  12. Datasets Statistics:
  13. Evaluation Metrics:
  14. Baseline Models:
  15. Parameter Settings:
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Illustration of modal interference and our GRAM-based alignment framework. (a) shows that when aligning the entity Anne Hathaway, the visual and attribute features of the entity Kirsten Dunst introduce interference to the task. (b) shows how Gram-based Loss imposes global constraints over cross-modal feature distributions.
  • Figure 2: The overall framework of MyGram. (a) Multi-modal Feature Extraction: Extract uni-modal embeddings for each entity from different modalities; (b) Modality-aware Diffusion Learning: Enhance modality features with structural contextual information; (c) Multi-modal training and Learning: employing gram loss to establish alignment between equivalent entities.
  • Figure 3: Ablation study on key components of proposed MyGram on (a) FBDB15K and (b) FBYG15K.
  • Figure 4: Low resource performance comparison on (a) FBDB15K and (b) FBYG15K.
  • Figure 5: A case example of multi-modal entity alignment task for entity Shang Hai.