Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

TL;DR

This paper extends the LAMA knowledge-probing paradigm to a multilingual setting by translating TREx and GoogleRE into 53 languages (mLAMA) and evaluating mBERT as a cross-lingual knowledge base. It introduces Typed Querying (TyQ) to constrain predictions to relation-appropriate candidate sets and handles multitoken objects, showing that TyQ outperforms untyped approaches and that pooling predictions across languages yields substantial gains. The findings reveal language-dependent knowledge representations and biases in mBERT, while pooling can even surpass monolingual English baselines, underscoring the value of multilingual probing. The authors provide a new dataset and code to enable widespread multilingual knowledge assessment beyond English, with implications for cross-lingual reasoning and bias analysis.

Abstract

Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

Figures (6)

• Figure 1: x-axis is the number of translated triples, y-axis the number of languages. There are 39,567 triples in the original LAMA (TREx and GoogleRE).
• Figure 2: Distribution of p1 scores for 53 languages in UnTyQ vs. TyQ. Left: singletoken (object $=$ 1 token). Right: multitoken (object $>$ 1 token).
• Figure 3: p1 of BERT (red) vs mBERT[x] (blue) divided by p1 of mBERT[en] on the same set of triples in each language x. mBERT captures less factual knowledge than monolingual English BERT. While performance is reasonable for 21 languages, it is below 60% for 32 languages. Dashed line is rel-p1 of mBERT[en] (by definition equal to 1.0). Performance of BERT varies slightly as the set of triples is different for each language. Note that the Wikipedia of Cebuano (ceb) consists mostly of machine translated articles.
• Figure 4: Three randomly sampled data entries from mLAMA per language. Due to the automatic generation of the dataset not all of them are fully correct.
• Figure 5: Data samples continued.