Tracing Multilingual Factual Knowledge Acquisition in Pretraining

TL;DR

This study traces how multilingual factual knowledge and crosslingual consistency emerge during pretraining of an English-centric decoder model (OLMo-7B) using KLAR prompts over 12 languages and 1,197 facts. It shows that factual recall develops rapidly in early pretraining and that fact frequency in the pretraining corpus largely explains recall accuracy across languages, with crosslingual transfer from English enabling some low-frequency non-English facts, particularly for named-entity relations. The authors identify two acquisition pathways—frequency-driven learning (dominant and language-agnostic) and crosslingual transfer (limited but present in early training for certain relations)—and demonstrate that script similarity influences transfer more than deep language relatedness. They provide a resource package (code and data) to support further research and highlight implications for understanding multilingual knowledge in pretraining and crosslingual transfer dynamics.

Abstract

Large Language Models (LLMs) are capable of recalling multilingual factual knowledge present in their pretraining data. However, most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency throughout pretraining largely unexplored. In this work, we trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study. We find that both accuracy and consistency improve over time for most languages. We show that this improvement is primarily driven by the fact frequency in the pretraining corpus: more frequent facts are more likely to be recalled correctly, regardless of language. Yet, some low-frequency facts in non-English languages can still be correctly recalled. Our analysis reveals that these instances largely benefit from crosslingual transfer of their English counterparts -- an effect that emerges predominantly in the early stages of pretraining. We pinpoint two distinct pathways through which multilingual factual knowledge acquisition occurs: (1) frequency-driven learning, which is dominant and language-agnostic, and (2) crosslingual transfer, which is limited in scale and typically constrained to relation types involving named entities. We release our code and data to facilitate further research at https://github.com/cisnlp/multilingual-fact-tracing.

Figures (29)

• Figure 1: Relationship between fact frequency and factual recall in Catalan. High-frequency facts are more likely to be correctly recalled, indicating the effect of frequency-based learning. Meanwhile, the correct recall of some low-frequency facts suggests the influence of crosslingual transfer from other languages.
• Figure 2: Factual accuracy (ACC) and crosslingual consistency (CO). While factual knowledge is rapidly acquired during the early stages of pretraining and is reasonably high in many languages, a substantial performance gap remains between English and most other languages, highlighting the limitations of crosslingual knowledge transfer.
• Figure 3: Relationship between fact frequency and the probability of correct factual recall. A consistent upward trend across individual languages indicates that higher-frequency facts are more likely to be recalled by the model.
• Figure 4: Relationship between fact frequency and factual recall for all languages and six pretraining checkpoints. High-frequency facts are more likely to be correctly recalled than rare ones. This frequency-correctness correlation emerges early in pretraining and becomes more pronounced over time.
• Figure 5: Dynamics of learning for SCLFPs (surprisingly correct low frequency predictions, i.e., FNs in Table \ref{['tab:frequency_threshold']}) across 8 languages. Crosslingual transfer emerges early in pretraining and continues to strengthen over time.