Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval-augmented generation in multilingual settings

Nadezhda Chirkova, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, Vassilina Nikoulina

TL;DR

This work systematically investigates retrieval-augmented generation in multilingual settings (mRAG) across 13 languages, building a public baseline pipeline on BERGEN and evaluating with MKQA and XOR-TyDi QA datasets. It demonstrates that strong multilingual retrievers (BGE-m3) and multilingual-by-design generators (notably Command-R-35B) are essential, and that careful prompting (including translating the system prompt and forcing generation in the user language) significantly boosts performance. The authors introduce a multilingual evaluation approach using character 3-gram recall to handle transliteration and language variation, and show that retrieving from multilingual Wikipedia often yields the best results. They also highlight limitations such as code-switching and fluency issues, offering concrete directions for future work and releasing the code for reproducibility and further research.

Abstract

Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. In this work, we consider RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages, and investigate which components and with which adjustments are needed to build a well-performing mRAG pipeline, that can be used as a strong baseline in future works. Our findings highlight that despite the availability of high-quality off-the-shelf multilingual retrievers and generators, task-specific prompt engineering is needed to enable generation in user languages. Moreover, current evaluation metrics need adjustments for multilingual setting, to account for variations in spelling named entities. The main limitations to be addressed in future works include frequent code-switching in non-Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval. We release the code for the resulting mRAG baseline pipeline at https://github.com/naver/bergen.

Retrieval-augmented generation in multilingual settings

TL;DR

This work systematically investigates retrieval-augmented generation in multilingual settings (mRAG) across 13 languages, building a public baseline pipeline on BERGEN and evaluating with MKQA and XOR-TyDi QA datasets. It demonstrates that strong multilingual retrievers (BGE-m3) and multilingual-by-design generators (notably Command-R-35B) are essential, and that careful prompting (including translating the system prompt and forcing generation in the user language) significantly boosts performance. The authors introduce a multilingual evaluation approach using character 3-gram recall to handle transliteration and language variation, and show that retrieving from multilingual Wikipedia often yields the best results. They also highlight limitations such as code-switching and fluency issues, offering concrete directions for future work and releasing the code for reproducibility and further research.

Abstract

Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. In this work, we consider RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages, and investigate which components and with which adjustments are needed to build a well-performing mRAG pipeline, that can be used as a strong baseline in future works. Our findings highlight that despite the availability of high-quality off-the-shelf multilingual retrievers and generators, task-specific prompt engineering is needed to enable generation in user languages. Moreover, current evaluation metrics need adjustments for multilingual setting, to account for variations in spelling named entities. The main limitations to be addressed in future works include frequent code-switching in non-Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval. We release the code for the resulting mRAG baseline pipeline at https://github.com/naver/bergen.
Paper Structure (21 sections, 1 figure, 8 tables)

This paper contains 21 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual RAG pipeline
  4. Multilingual retrievers.
  5. Multilingual generation.
  6. System prompt.
  7. Multilingual QA datasets.
  8. Evaluation.
  9. Experimental details
  10. Retrieval.
  11. Generation.
  12. Evaluation.
  13. Results and discussion
  14. Prompting strategy: importance of translating the system prompt into target languages and specifying the desired language of the response.
  15. Generator model: importance of using a strong multilingual base model.
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Multilingual retrieval-augmented generation pipeline. We study which components are required to build a well performing mRAG pipeline, that can be used as a strong baseline in future works.