Towards Structure-aware Model for Multi-modal Knowledge Graph Completion

TL;DR

The paper tackles multi-modal knowledge graph completion (MMKGC) by emphasizing the structural backbone while enabling fine-grained, cross-modal interactions. It introduces TSAM, a structure-aware model that combines Fine-grained Modality Awareness Fusion (FgMAF) and Structure-aware Contrastive Learning (SaCL) to align visual and textual modalities with the graph structure. Across three public benchmarks, TSAM achieves state-of-the-art results, with notable improvements in MRR and Hits@1, by reducing cross-modal noise and maintaining structural dominance. The work highlights the practical impact of integrating structured information with multimodal cues for robust MMKGC, suggesting avenues for scalability and deeper integration with large language models.

Abstract

Knowledge graphs (KGs) play a key role in promoting various multimedia and AI applications. However, with the explosive growth of multi-modal information, traditional knowledge graph completion (KGC) models cannot be directly applied. This has attracted a large number of researchers to study multi-modal knowledge graph completion (MMKGC). Since MMKG extends KG to the visual and textual domains, MMKGC faces two main challenges: (1) how to deal with the fine-grained modality information interaction and awareness; (2) how to ensure the dominant role of graph structure in multi-modal knowledge fusion and deal with the noise generated by other modalities during modality fusion. To address these challenges, this paper proposes a novel MMKGC model named TSAM, which integrates fine-grained modality interaction and dominant graph structure to form a high-performance MMKGC framework. Specifically, to solve the challenges, TSAM proposes the Fine-grained Modality Awareness Fusion method (FgMAF), which uses pre-trained language models to better capture fine-grained semantic information interaction of different modalities and employs an attention mechanism to achieve fine-grained modality awareness and fusion. Additionally, TSAM presents the Structure-aware Contrastive Learning method (SaCL), which utilizes two contrastive learning approaches to align other modalities more closely with the structured modality. Extensive experiments show that the proposed TSAM model significantly outperforms existing MMKGC models on widely used multi-modal datasets.

Figures (7)

• Figure 1: A simple example of MMKGC, which includes not only the structural modality but also visual and textual modalities. There are fine-grained interactions between modalities; for instance, the different modalities linked by various dashed lines represent semantically similar meanings at a fine-grained level.
• Figure 2: Performance of the MyGOzhang2024mygo and OTKGEcao2022otkge models in terms of MRR and Hits@1 after completely removing all modality knowledge.
• Figure 3: The Architecture of the TSAM Model. TSAM incorporates the FgMAF method to better fuse and perceive various modalities knowledge in MMKG, while the SaCL method is employed to align other modal knowledge with the structural modality, with the structural modality as the dominant factor. TSAM employs iterative entity representation updates and contrastive learning to achieve representation learning. The optimized entity and relation representations are then input into the scoring function to perform relevant triplet link prediction.
• Figure 4: The experimental results of the TSAM model using Bert-base/large, RoBERTa- large, and DeBERTa-base/large as decoders on the DB15K and MKG-W datasets.
• Figure 5: Parameter sensitivity experiment of the number of Temperature parameter $\tau$