Context Volume Drives Performance: Tackling Domain Shift in Extremely Low-Resource Translation via RAG

TL;DR

This work introduces a hybrid framework where a fine-tuned NMT model generates an initial draft, which is then refined by a Large Language Model (LLM) using Retrieval-Augmented Generation (RAG), effectively matching the original in-domain quality.

Abstract

Neural Machine Translation (NMT) models for low-resource languages suffer significant performance degradation under domain shift. We quantify this challenge using Dhao, an indigenous language of Eastern Indonesia with no digital footprint beyond the New Testament (NT). When applied to the unseen Old Testament (OT), a standard NMT model fine-tuned on the NT drops from an in-domain score of 36.17 chrF++ to 27.11 chrF++. To recover this loss, we introduce a hybrid framework where a fine-tuned NMT model generates an initial draft, which is then refined by a Large Language Model (LLM) using Retrieval-Augmented Generation (RAG). The final system achieves 35.21 chrF++ (+8.10 recovery), effectively matching the original in-domain quality. Our analysis reveals that this performance is driven primarily by the number of retrieved examples rather than the choice of retrieval algorithm. Qualitative analysis confirms the LLM acts as a robust "safety net," repairing severe failures in zero-shot domains.

Figures (7)

• Figure 1: Hybrid Post-Editing Architecture. The workflow integrates two parallel streams. Center: The NLLB model, fine-tuned on the In-Domain (NT) corpus, generates an initial Translated Source Text (draft). Left: A Retrieval-Augmented Generation (RAG) module queries the combined corpus (Grammar Book + NT) to extract relevant examples for the Few-shot Prompt. Bottom: The LLM Post-Editor (Gemini) synthesizes the NMT draft, the original source, and the retrieved context to produce the final Post-edited Translation. Legend: Orange: Core objects (data and models) $|$ Green: Processing steps $|$ Red: Intermediate outputs $|$ Purple: Prompt configuration $|$ Gray: Final output.
• Figure 2: Impact of Context Volume on Performance. Comparison of absolute chrF++ scores across retrieval strategies relative to the Fixed 5-Shot Baseline (dashed red line, corresponding to the LLM Post-Editing baseline in Table \ref{['tab:baseline_performance']}). Left: Sentence-level methods (BGE, BM25, ChrF-RAG) show rapid initial gains but plateau at $K \approx 60$. Right: The Word-Level strategy allows the model to ingest a higher effective volume of examples (peaking at effective $K \approx 137$) to squeeze out marginal performance gains.
• Figure 3: Retrieval Strategy Performance Convergence. We compare the optimal configuration of the Word-Level strategy (Fuzzy Matching, top-10) against the best configurations of three sentence-level baselines: ChrF-RAG ($k=60$), BGE Semantic Retrieval ($k=100$), and BM25 ($k=80$). The bar chart displays the absolute chrF++ scores, with the dashed red line indicating the Fixed 5-shot Baseline.
• Figure 4: Lexicon Retrieval Performance. Impact of increasing the number of retrieved lexicon entries ($K$) on post-editing performance. Unlike sentence-level retrieval which plateaus, lexicon retrieval improves monotonically with volume. The highest performance is achieved by providing the Full Lexicon (star marker), yielding a chrF++ score of 31.32.
• Figure 5: The Out-of-Vocabulary (OOV) rate of the in-domain NT Validation set (8.1%) versus the out-of-domain OT Test set (25.9%). All rates are calculated relative to the NT training vocabulary.