On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation

TL;DR

This study disentangles how multilingual LLMs use cross-lingual context in retrieval-augmented generation by controlling retrieval quality and evaluating single- and multi-passage setups across XQUAD, MKQA, and GMMLU. It finds that LLMs can extract information from passages in other languages, but struggle to generate answers in the query language, indicating a decoding bottleneck rather than a lack of contextual understanding. Interpretability analyses (MIRAGE) confirm content-driven predictions and reveal that distractors in context harm open-domain QA across languages, stressing the importance of passage ranking. The results support the value of cross-lingual retrieval for diverse languages while highlighting the need to improve language-specific generation and robust distractor handling for more reliable mRAG systems.

Abstract

Retrieval-augmented generation (RAG) with large language models (LLMs) has demonstrated strong performance in multilingual question-answering (QA) tasks by leveraging relevant passages retrieved from corpora. In multilingual RAG (mRAG), the retrieved passages can be written in languages other than that of the query entered by the user, making it challenging for LLMs to effectively utilize the provided information. Recent research suggests that retrieving passages from multilingual corpora can improve RAG performance, particularly for low-resource languages. However, the extent to which LLMs can leverage different kinds of multilingual contexts to generate accurate answers, *independently from retrieval quality*, remains understudied. In this paper, we conduct an extensive assessment of LLMs' ability to (i) make consistent use of a relevant passage regardless of its language, (ii) respond in the expected language, and (iii) focus on the relevant passage even when multiple `distracting' passages in different languages are provided in the context. Our experiments with four LLMs across three QA datasets covering a total of 48 languages reveal a surprising ability of LLMs to extract the relevant information from passages in a different language than the query, but a much weaker ability to formulate a full answer in the correct language. Our analysis, based on both accuracy and feature attribution techniques, further shows that distracting passages negatively impact answer quality regardless of their language. However, distractors in the query language exert a slightly stronger influence. Taken together, our findings deepen the understanding of how LLMs utilize context in mRAG systems, providing directions for future improvements.

Figures (4)

• Figure 1: Illustration of the contributions and proposed assessment frameworks of this paper.
• Figure 2: Performance on XQUAD, MKQA, GMMLU-Open, and GMMLU-Choice, where the LLMs are provided with no retrieved passage or one gold passage in either in-language or out-language. The shading on the bars represents the ratio of questions that can be correctly answered but in the wrong passage language, which does not apply to GMMLU-Choice since the evaluation on it is not affected by the generation language.
• Figure 3: Answer accuracy (%) on XQUAD among different query-passage language combinations. Only model answers in the correct (i.e., query) language are considered as correct.
• Figure 4: Model performance on XQUAD when the query is concatenated with passage in each studied language. Top: The portion of queries that can be correctly answered in the user language. Bottom: The portion of queries for which the LLMs generate the correct answer but in the wrong (passage) language. For a part of correctly answered queries, the gold answers are the same words in the passage and query languages. In these cases, we only consider them in the above heatmaps to ensure that there is no overlapping between the two vertical heatmaps and that they are addable.