Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Factual Consistency of Multilingual Pretrained Language Models

Constanza Fierro, Anders Søgaard

TL;DR

This work investigates the factual knowledge consistency of multilingual pretrained language models by introducing mParaRel, a multilingual extension of ParaRel across 46 languages. Using cloze-style probing on mBERT and XLM-R, the authors show that consistency is already fragile in English and becomes substantially worse in other languages, even when model accuracy varies. The methodology combines automated translations from multiple MT systems with a targeted human review and a robust evaluation framework, including multi-token object predictions. The findings highlight critical limitations for leveraging multilingual PLMs as reliable knowledge bases and provide a resource and benchmarks for future cross-language analysis and improvement of factual recall in multilingual NLP systems.

Abstract

Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.

Factual Consistency of Multilingual Pretrained Language Models

TL;DR

This work investigates the factual knowledge consistency of multilingual pretrained language models by introducing mParaRel, a multilingual extension of ParaRel across 46 languages. Using cloze-style probing on mBERT and XLM-R, the authors show that consistency is already fragile in English and becomes substantially worse in other languages, even when model accuracy varies. The methodology combines automated translations from multiple MT systems with a targeted human review and a robust evaluation framework, including multi-token object predictions. The findings highlight critical limitations for leveraging multilingual PLMs as reliable knowledge bases and provide a resource and benchmarks for future cross-language analysis and improvement of factual recall in multilingual NLP systems.

Abstract

Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.
Paper Structure (13 sections, 1 equation, 2 figures, 2 tables)

This paper contains 13 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Probing Consistency
  4. mParaRel
  5. Translations
  6. Human Evaluation
  7. Experiments
  8. Querying Language Models
  9. Evaluation
  10. Results and Discussion
  11. Consistency across languages
  12. Related Work
  13. Conclusion

Figures (2)

  • Figure 1: Number of examples per language. Manually reviewed languages are underlined. The order is given by the consistency results (see Figure \ref{['fig:mpararel_results']}).
  • Figure 2: mBERT and XLM-R results on mParaRel after and before the human review (\ref{['sec:human_evaluation']}). The order of the languages follows the consistency results in mBERT, and the languages underlined were manually reviewed.