XRAG: Cross-lingual Retrieval-Augmented Generation

TL;DR

XRAG introduces a cross-lingual RAG benchmark built from News Crawl data to evaluate how LLMs generate answers when the question language differs from the retrieved documents. A novel generation workflow creates natural, cross-document QA pairs that require multi-source reasoning, with rigorous human quality control and translations across German, Spanish, Chinese, and Arabic. Evaluations across five LLMs reveal that models struggle with response language correctness in monolingual retrieval and with cross-language reasoning in multilingual retrieval, underscoring a gap to human performance and highlighting XRAG as a robust testbed for cross-lingual reasoning in retrieval-augmented generation.

Abstract

We propose XRAG, a novel benchmark designed to evaluate the generation abilities of LLMs in cross-lingual Retrieval-Augmented Generation (RAG) settings where the user language does not match the retrieval results. XRAG is constructed from recent news articles to ensure that its questions require external knowledge to be answered. It covers the real-world scenarios of monolingual and multilingual retrieval, and provides relevancy annotations for each retrieved document. Our novel dataset construction pipeline results in questions that require complex reasoning, as evidenced by the significant gap between human and LLM performance. Consequently, XRAG serves as a valuable benchmark for studying LLM reasoning abilities, even before considering the additional cross-lingual complexity. Experimental results on five LLMs uncover two previously unreported challenges in cross-lingual RAG: 1) in the monolingual retrieval setting, all evaluated models struggle with response language correctness; 2) in the multilingual retrieval setting, the main challenge lies in reasoning over retrieved information across languages rather than generation of non-English text.

Figures (24)

• Figure 1: Two cases of cross-lingual RAG: (a) monolingual retrieval, where the LLM uses retrieved English documents to respond to a German query; (b) multilingual retrieval, where the LLM uses retrieved English and German documents to respond to a German query.
• Figure 2: Each instance $(q, a, {\rm D}^+, {\rm D}^-)$ in XRAG — where $q$ is the question, $a$ the gold answer, $\rm D^+$ the supporting articles, and $\rm D^-$ the distractors — is constructed as follows: (1) find two related articles; (2) generate an English cross-document Q&A pair using the two articles; (3) evaluate the quality of the Q&A pair; (4) translate the Q&A pair into language X $\in$ {German, Spanish, Chinese, Arabic}; and (5) collect distracting articles for the question.
• Figure 3: LLM-based workflow for generating English cross-document questions from a pair of related articles: (1) generate a summary for each article; (2) create simple English Q&A pairs from each summary that require only one-step reasoning; (3) identify connections between the two sets of Q&A pairs, select related ones, and construct a new Q&A pair that requires reasoning across multiple pieces of information from the selected pairs and summaries.
• Figure 4: Evaluation workflow on XRAG: (1) the evaluated LLM generates a response $\tilde{a}$ for a question $q$ based on two supporting articles ${\rm D}^{+}$, and six distracting articles ${\rm D}^{-}$; (2) the response is checked for language correctness; (3) a panel of three LLM judges independently assess the factual accuracy of the response based on the question $q$ and a gold answer $a$, with the final judgment based on majority vote; (4) the final evaluation combines the factual judgment and language correctness.
• Figure 5: Percentage of instances in cross-lingual RAG with monolingual retrieval (English documents) where LLMs respond in English instead of the German or Chinese question language.