ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer

TL;DR

ECLeKTic introduces a multilingual closed-book QA benchmark designed to evaluate cross-lingual knowledge transfer in LLMs by targeting facts present in one language’s Wikipedia but absent in others. The dataset comprises 4,608 QA pairs across 12 languages (with 4,224 target and 384 source examples), created through a two-stage human-assisted process and translated across languages for robust cross-language testing. Evaluation hinges on judge-based correctness and a string-matching baseline, revealing that even state-of-the-art models struggle to transfer knowledge across languages, with the best overall performance around 41.6% and transfer gains limited by linguistic script and prompt design. Ablation studies show that providing correct source-language context dramatically boosts cross-lingual success, while mere hints have limited impact, underscoring retrieval as the core bottleneck. The work highlights significant headroom for improving multilingual knowledge sharing and provides a reproducible, black-box framework for assessing API-restricted models.

Abstract

To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired.

Figures (13)

• Figure 1: A model incapable of cross-lingual knowledge transfer (middle box) can only answer factual questions in their source language, that is, the language in which the information appeared in its training. It cannot answer the same question when translated into another target language. A transfer capable model (bottom box), is able to answer questions no matter the language. ECLeKTic allows distinguishing between the two by targeting facts that unevenly distributed in the model's training data.
• Figure 2: A schematic overview of the creation of ECLeKTic and its application in evaluating language models.
• Figure 3: Break down of the examples in ECLeKTic by source language.
• Figure 4: Break down of the examples in ECLeKTic by domains of knowledge. A breakdown done separately per source language can be found in \ref{['sec:per-language-domains']}.
• Figure 5: Transfer score results of Gemini 2.0 Pro broken down per source and target language. blue is better, red is worse. Note the diagonal is perfect by the definition of the transfer metric.