Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models

TL;DR

The paper tackles cross-lingual consistency of factual knowledge in multilingual PLMs by introducing RankC, a metric that measures consistency across languages independently of accuracy. It constructs BMLAMA, a Balanced Multilingual LAMA benchmark, to enable fair cross-language comparison and evaluates encoder-only, decoder-only, and encoder-decoder models, including XLM-RoBERTa, mT5, and BLOOM variants. The study finds overall low cross-lingual consistency, with model size providing little improvement, and identifies subword vocabulary overlap as a strong predictor of RankC, suggesting knowledge percolation occurs primarily through shared embeddings rather than deeper language-agnostic representations. A case study using ROME shows that inserted knowledge propagates more readily to languages with high RankC with the source language, highlighting implications for multilingual model editing and knowledge incorporation.

Abstract

Multilingual large-scale Pretrained Language Models (PLMs) have been shown to store considerable amounts of factual knowledge, but large variations are observed across languages. With the ultimate goal of ensuring that users with different language backgrounds obtain consistent feedback from the same model, we study the cross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs. To this end, we propose a Ranking-based Consistency (RankC) metric to evaluate knowledge consistency across languages independently from accuracy. Using this metric, we conduct an in-depth analysis of the determining factors for CLC, both at model level and at language-pair level. Among other results, we find that increasing model size leads to higher factual probing accuracy in most languages, but does not improve cross-lingual consistency. Finally, we conduct a case study on CLC when new factual associations are inserted in the PLMs via model editing. Results on a small sample of facts inserted in English reveal a clear pattern whereby the new piece of knowledge transfers only to languages with which English has a high RankC score.

Figures (11)

• Figure 1: Motivating example: Some languages share consistent knowledge in the multilingual PLM BLOOM-3b, while others do not.
• Figure 2: Factual knowledge probing accuracy (%) in various multilingual PLMs, measured on BMLAMA-17.
• Figure 3: Knowledge consistency (RankC %) between language pairs in the PLMs, with darker shading denoting higher-consistency pairs. Green bars: Probing accuracy of each language.
• Figure 4: Cross-lingual consistency (RankC) shows little fluctuation among PLMs of different scales. The average probing accuracy of the four models is 25.97%, 24.77%, 22.93%, and 21.14%, respectively.
• Figure 5: Linear regression between subword vocabulary overlap (Flores) and CLC measured on BMLAMA-53 for BLOOM-3b. For results on mT5-large and XLM-RoBERTa-large, see Appendix \ref{['sec: appendix-probing53']}.