Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation

TL;DR

This work addresses incomplete multimodal knowledge graphs by introducing MyGO, a framework that tokenizes multi-modal entity data into fine-grained tokens and uses a cross-modal entity encoder within a hierarchical triple modeling architecture to predict missing triples. A Tucker-style relational decoder scores triples, while a fine-grained contrastive learning objective enhances representation distinctiveness. Empirical results on three public MMKGC benchmarks demonstrate state-of-the-art performance against 19 baselines, with notable gains in accuracy-sensitive metrics like Hit@1 and MRR, and analyses reveal robustness to tokenization choices and scalability with more multimodal data. The approach offers a scalable, interpretable path to leveraging fine-grained multimodal signals for improved knowledge graph completion, with practical implications for downstream multimodal reasoning tasks.

Abstract

Multi-modal knowledge graph completion (MMKGC) aims to discover unobserved knowledge from given knowledge graphs, collaboratively leveraging structural information from the triples and multi-modal information of the entities to overcome the inherent incompleteness. Existing MMKGC methods usually extract multi-modal features with pre-trained models, resulting in coarse handling of multi-modal entity information, overlooking the nuanced, fine-grained semantic details and their complex interactions. To tackle this shortfall, we introduce a novel framework MyGO to tokenize, fuse, and augment the fine-grained multi-modal representations of entities and enhance the MMKGC performance. Motivated by the tokenization technology, MyGO tokenizes multi-modal entity information as fine-grained discrete tokens and learns entity representations with a cross-modal entity encoder. To further augment the multi-modal representations, MyGO incorporates fine-grained contrastive learning to highlight the specificity of the entity representations. Experiments on standard MMKGC benchmarks reveal that our method surpasses 19 of the latest models, underlining its superior performance. Code and data can be found in https://github.com/zjukg/MyGO

Figures (7)

• Figure 1: An intuition of existing MMKGC methods and MyGO. MyGO attempts to tokenize raw modality data into fine-grained tokens and learn the fine-grained entity representations to model semantic unit interactions.
• Figure 2: The overview of our MyGO framework. We mainly have three parts of new designs in MyGO to tokenize, fuse, and augment the fine-grained multi-modal semantic information in the MMKGs.
• Figure 3: The MMKGC results of MyGO and several baselines with different numbers of entity images.
• Figure 4: The MRR results with different modality token amount $m$ and $n$ on DB15K.
• Figure 5: MMKGC results using different tokenizers.