Quality-Aware Translation Tagging in Multilingual RAG system

TL;DR

The paper tackles translation quality in multilingual retrieval-augmented generation by introducing Quality-Aware Translation Tagging (QTT-RAG), which evaluates translated passages on semantic equivalence, grammatical accuracy, and naturalness and attaches these scores as metadata rather than rewriting content. This metadata guides in-context generation to rely more on high-quality translations, reducing factual distortion common in prior methods like CrossRAG and DKM-RAG. Through experiments on XOR-TyDi QA and MKQA across Korean, Finnish, and Chinese with six instruction-tuned LLMs, QTT-RAG achieves higher robustness and improved character-level recall than baselines, especially in low-resource languages. The approach balances translation usage with content fidelity, offering a practical strategy for leveraging cross-lingual documents in multilingual QA while exposing metadata to the generator for better decision-making.

Abstract

Multilingual Retrieval-Augmented Generation (mRAG) often retrieves English documents and translates them into the query language for low-resource settings. However, poor translation quality degrades response generation performance. Existing approaches either assume sufficient translation quality or utilize the rewriting method, which introduces factual distortion and hallucinations. To mitigate these problems, we propose Quality-Aware Translation Tagging in mRAG (QTT-RAG), which explicitly evaluates translation quality along three dimensions-semantic equivalence, grammatical accuracy, and naturalness&fluency-and attach these scores as metadata without altering the original content. We evaluate QTT-RAG against CrossRAG and DKM-RAG as baselines in two open-domain QA benchmarks (XORQA, MKQA) using six instruction-tuned LLMs ranging from 2.4B to 14B parameters, covering two low-resource languages (Korean and Finnish) and one high-resource language (Chinese). QTT-RAG outperforms the baselines by preserving factual integrity while enabling generator models to make informed decisions based on translation reliability. This approach allows for effective usage of cross-lingual documents in low-resource settings with limited native language documents, offering a practical and robust solution across multilingual domains.

Figures (2)

• Figure 1: This figure illustrates a failure case of the previous approach (DKM-RAG). Hallucination arises when the LLM rewrites the translated documents, causing the generator to eventually produce an incorrect answer.
• Figure 2: Overview of QTT-RAG System. After retrieving and reranking the top-5 most relevant documents, documents originally written in the query language (KO) are passed directly to the generator, whereas documents in foreign languages (EN) are translated and automatically scored along three dimensions: semantic equivalence (E), grammatical accuracy (G), and naturalness & fluency (N). These passages are then re-inserted with the corresponding quality tags. The generator receives this quality-aware, tagged input, enabling it to produce factually grounded and translation-sensitive responses.