Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints

TL;DR

The paper tackles multilingual knowledge graph completion (mKGC) by addressing English-centric bias in pretrained language models through the introduction of global and local knowledge constraints. It proposes a Triple Encoder framework with a global translational constraint based on $\mathbf{h}_{[H]} + \mathbf{h}_{[R]} \approx \mathbf{h}_{[T]}$ and a local mutual information objective between query and tail representations, optimized via a Jensen–Shannon MI estimator, combined with a generation objective $\mathcal{L}= \mathcal{L}_G + \alpha\mathcal{L}_P + \beta\mathcal{L}_E$. The method yields significant gains over Prix-LM on seven language KG datasets, including average improvements of $12.32\%$ (Hits@1), $11.39\%$ (Hits@3), and $16.03\%$ (Hits@10), and显示 notable benefits for cross-lingual entity alignment and low-resource languages. These results demonstrate that incorporating structured knowledge constraints with PLMs enhances mKGC performance and reduces language/data bias, with practical impact for multilingual NLP applications.

Abstract

Multilingual Knowledge Graph Completion (mKGC) aim at solving queries like (h, r, ?) in different languages by reasoning a tail entity t thus improving multilingual knowledge graphs. Previous studies leverage multilingual pretrained language models (PLMs) and the generative paradigm to achieve mKGC. Although multilingual pretrained language models contain extensive knowledge of different languages, its pretraining tasks cannot be directly aligned with the mKGC tasks. Moreover, the majority of KGs and PLMs currently available exhibit a pronounced English-centric bias. This makes it difficult for mKGC to achieve good results, particularly in the context of low-resource languages. To overcome previous problems, this paper introduces global and local knowledge constraints for mKGC. The former is used to constrain the reasoning of answer entities, while the latter is used to enhance the representation of query contexts. The proposed method makes the pretrained model better adapt to the mKGC task. Experimental results on public datasets demonstrate that our method outperforms the previous SOTA on Hits@1 and Hits@10 by an average of 12.32% and 16.03%, which indicates that our proposed method has significant enhancement on mKGC.

Figures (5)

• Figure 1: The top part introduces unbalance language distribution for DBpedia. The low part shows the sampling comparison results of Prix-LM model and our method. The type of prediction entity and the correct answer are shown in brackets and red font, respectively. Our approach exhibits superior consistency and accuracy in generating answers.
• Figure 2: This figure illustrates the architecture of the complete model, which is composed of four main components: a query encoder, a global knowledge constraint, a local knowledge constraint, and an answer generation module. The global knowledge learn from representations of head and relation (navy blue). The local knowledge learn from representations of query words (light blue). We use different colors to represent entities and relation in each module for a triple.
• Figure 3: The operation mechanism of mask matrix during training process. The darker squares indicate that attention is allowed, while the lighter squares indicate that attention is suppressed.
• Figure 4: This figure presents a comparison of the performance of our method and baseline model on a set of case studies. The blue font is used to indicate that the predicted answer aligns with the golden answer type. The bold font in the predicted answer signifies the correct answer.
• Figure 5: The figure presents the results of the Hits@k evaluation metric for a mKGC task, focusing on answers of varying lengths. In order to facilitate a more straightforward analysis, the results are limited to those sets of lengths that have more than 100 occurrences.