Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion

TL;DR

This work tackles multi-modal knowledge graph completion (MMKGC) under modality imbalance and missing data. It introduces AdaMF-MAT, combining Adaptive Multi-modal Fusion (AdaMF) to learn entity-specific modality weights and Modality Adversarial Training (MAT) to generate synthetic multimodal embeddings via a generator–discriminator framework, enhancing learning from imbalanced information. Empirical results on three public MMKGC benchmarks show state-of-the-art performance against 19 baselines, with notable gains in strict metrics and demonstrated robustness to modality-missing scenarios. The approach advances practical MMKGC by improving how modality signals are fused and augmented during training, potentially benefiting downstream reasoning and downstream applications.

Abstract

Multi-modal knowledge graph completion (MMKGC) aims to predict the missing triples in the multi-modal knowledge graphs by incorporating structural, visual, and textual information of entities into the discriminant models. The information from different modalities will work together to measure the triple plausibility. Existing MMKGC methods overlook the imbalance problem of modality information among entities, resulting in inadequate modal fusion and inefficient utilization of the raw modality information. To address the mentioned problems, we propose Adaptive Multi-modal Fusion and Modality Adversarial Training (AdaMF-MAT) to unleash the power of imbalanced modality information for MMKGC. AdaMF-MAT achieves multi-modal fusion with adaptive modality weights and further generates adversarial samples by modality-adversarial training to enhance the imbalanced modality information. Our approach is a co-design of the MMKGC model and training strategy which can outperform 19 recent MMKGC methods and achieve new state-of-the-art results on three public MMKGC benchmarks. Our code and data have been released at https://github.com/zjukg/AdaMF-MAT.

Figures (5)

• Figure 1: A simple example to show that knowledge graph reasoning with limited modal information may lead to wrong prediction.
• Figure 2: Overview of our method AdaMF-MAT. The feature encoders are designed to encode different modal features (visual/textual/structural) respectively. Each FC represents a fully-connected projection layer. The adaptive multi-modal fusion module is designed to get the fused joint embedding adaptively. The modality adversarial training module employs generators to generate synthetic multi-modal embeddings to construct adversarial examples. The KGC decoder serves as the discriminator and will be enhanced by these adversarial examples during training.
• Figure 3: Link prediction results on modality-missing DB15K dataset. The x-axis represents the modality-missing ratios. We report the MRR results of three various MMKGC models (AdaMF, TBKGC, IKRL). The missing modal information is randomly initialized first, as commonly done in existing methods DBLP:conf/starsem/TBKGC. w/ MAT and w/o MAT indicate the MMKGC model trained with MAT and without MAT respectively.
• Figure 4: Parameter analysis results about the number of adversarial examples. We report the MRR and Hit@1 results.
• Figure 5: Adaptive weight visualization results of AdaMF for different relations. We consider the AdaMF models trained w/ and w/o MAT. We partitioned the test triples by relation and calculated the average modality weights among the entities.