Evaluating Contextually Mediated Factual Recall in Multilingual Large Language Models

TL;DR

This paper addresses how multilingual LLMs recall factual knowledge when the target entity is introduced through naturalistic context rather than direct queries. It proposes a contextually mediated prompt framework, using synthetic names to control biases and evaluating across five languages and three model families with the KLAR dataset. The results show a consistent degradation in factual recall due to contextual mediation, with the magnitude of degradation depending on the relation and model size—larger models are more robust. Real-name biases do not exhibit a systematic effect, suggesting context plays a dominant role in recall performance. The findings reveal a gap between isolated factual recall and context-dependent understanding, highlighting the need for context-aware, multilingual benchmarking of LLMs.

Abstract

Large language models (LLMs) can recall a wide range of factual knowledge across languages. However, existing factual recall evaluations primarily assess fact retrieval in isolation, where the queried entity is explicitly named and the fact is requested directly. In natural language use, facts are often accessed through context, where the relevant entity is introduced only indirectly. In this work, we study contextually mediated factual recall, asking whether LLMs can reliably retrieve factual knowledge when the target entity is embedded in a naturalistic context rather than queried explicitly, across languages. We construct controlled prompts that preserve the underlying fact while introducing referential mediation through contextual sentences. To disentangle contextual effects from name-specific associations, we further compare performance using synthetic names and real names across languages. Evaluating multiple model families in five languages, we find that contextual mediation consistently degrades factual recall, with substantial variation across relations. Larger models are more robust to contextual mediation, exhibiting a reduced performance gap relative to direct queries, while the effect of real names and name origin is mixed and unsystematic. These findings highlight a gap between isolated factual recall and context-dependent language understanding in multilingual LLMs.

Figures (3)

• Figure 1: Per-relation factual recall performance for five languages under direct factual queries and contextually mediated queries. Each radar plot shows accuracy averaged across all models. Across languages, introducing contextual mediation generally leads to a relation-dependent performance degradation compared to direct queries.
• Figure 2: Average factual recall performance across relations as a function of model size for each model family. Larger models in the LLaMA and Gemma families exhibit increased robustness to contextual mediation, while the Qwen family shows a weaker size effect.
• Figure 3: Per-language comparison of contextually mediated factual recall using synthetic versus real names, averaged across relations for each model. The effect of replacing synthetic names with real names is highly mixed across both languages and models, showing no consistent increase or decrease in performance.