Language Models' Factuality Depends on the Language of Inquiry

TL;DR

The paper reveals a cross-lingual factual knowledge transfer gap in multilingual language models, showing that country-specific facts are often recalled only in the language in which they are learned. It introduces a 13-language benchmark with three tasks—Factual Recall, In-context Recall, and Counter-Factual Context Adherence—and defines three metrics (FRS, KTS, X-FaKT) to quantify factual recall and cross-lingual transfer. Results indicate that model size and resource level influence cross-lingual performance, with larger models and high-resource languages achieving better scores, yet persistent asymmetries and a trade-off with context adherence remain. The authors advocate calibrated multilingualism and provide an open benchmark to advance research in multilingual knowledge transfer, while cautioning about evaluator biases when using LMs as judges.

Abstract

Multilingual language models (LMs) are expected to recall factual knowledge consistently across languages, yet they often fail to transfer knowledge between languages even when they possess the correct information in one of the languages. For example, we find that an LM may correctly identify Rashed Al Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails to do so when asked in English or Swahili. To systematically investigate this limitation, we introduce a benchmark of 10,000 country-related facts across 13 languages and propose three novel metrics: Factual Recall Score, Knowledge Transferability Score, and Cross-Lingual Factual Knowledge Transferability Score-to quantify factual recall and knowledge transferability in LMs across different languages. Our results reveal fundamental weaknesses in today's state-of-the-art LMs, particularly in cross-lingual generalization where models fail to transfer knowledge effectively across different languages, leading to inconsistent performance sensitive to the language used. Our findings emphasize the need for LMs to recognize language-specific factual reliability and leverage the most trustworthy information across languages. We release our benchmark and evaluation framework to drive future research in multilingual knowledge transfer.

Figures (28)

• Figure 1: Illustratation of the cross-lingual factual knowledge transferability issue across linguistic knowledge clouds in LMs. The model correctly recalls that Rashed Al Shashai is from Saudi Arabia when queried in Arabic, but fails to retrieve this fact in English and Swahili, highlighting that factual knowledge is often stored in language-specific silos.
• Figure 2: Examples from our multilingual dataset illustrating three tasks. Factual Recall: LMs recall country-specific facts better in native languages, as seen with Dharan's correct identification in Nepali but incorrect in English. Incontext Recall: Models struggle with contextual reasoning, showing regional bias when associating names with countries. Counter-Factual Context Adherence: When given counterfactual prompts about well-known figures, models rely on prior knowledge, affecting their ability to adhere to provided context.
• Figure 3: Error rates for each model on the Factual Recall task. A clear pattern emerges, showing a decline in performance as we move from larger to smaller models (top to bottom) and from high-resource to low-resource languages (left to right).
• Figure 4: This figure illustrates the model-wise comparison of X-FAKT scores grouped by language families. A clear trend emerges, showing that as the model size increases within a family, the X-FAKT score tends to increase.
• Figure 5: Error rate for each model on Counter-Factual Context Adherence task. Models show high error rates in high resource languages such as English and French where they have high factual recall.