Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchu

TL;DR

The paper addresses the challenge of machine translation for low-resource languages by systematically analyzing how in-context information from dictionaries, parallel examples, grammar, and prompt strategies influences translation quality for Manchu. It employs a morphologically aware, sequential prompting pipeline and an encipherment experiment to separate pretrained knowledge from in-context learning, finding that high-quality dictionaries and closely related parallel examples yield the largest gains, while grammar and Chain-of-Thought prompting offer limited or negative impact. The study further demonstrates the value of in-context MT as a data-augmentation tool, showing that synthetic parallel data generated via prompts can substantially boost traditional NMT performance, enabling a compact model like mT5-small to approach large-LM results. Together, these findings illuminate how to leverage linguistic resources efficiently for low-resource MT and offer a practical route to building stronger MT systems with limited parallel data. The results have broad implications for deploying MT in endangered languages, suggesting concrete prompting strategies and data-augmentation workflows that scale with monolingual resources.

Abstract

In-context machine translation (MT) with large language models (LLMs) is a promising approach for low-resource MT, as it can readily take advantage of linguistic resources such as grammar books and dictionaries. Such resources are usually selectively integrated into the prompt so that LLMs can directly perform translation without any specific training, via their in-context learning capability (ICL). However, the relative importance of each type of resource, e.g., dictionary, grammar book, and retrieved parallel examples, is not entirely clear. To address this gap, this study systematically investigates how each resource and its quality affect the translation performance, with the Manchu language as our case study. To remove any prior knowledge of Manchu encoded in the LLM parameters and single out the effect of ICL, we also experiment with an enciphered version of Manchu texts. Our results indicate that high-quality dictionaries and good parallel examples are very helpful, while grammars hardly help. In a follow-up study, we showcase a promising application of in-context MT: parallel data augmentation as a way to bootstrap a conventional MT model. When monolingual data abound, generating synthetic parallel data through in-context MT offers a pathway to mitigate data scarcity and build effective and efficient low-resource neural MT systems.

Figures (3)

• Figure 1: Illustration of the in-context MT pipeline with the components of $\pi$, i.e., $\mu(\mathbf{x})$, $\mathrm{D}$, $\mathrm{P}$ and $\mathrm{G}$.
• Figure 2: Performance comparison between enciphered $\pi(\mu(\mathbf{x})_{e}, \mathrm{D^{l+s+c}_{e}}, \mathrm{P^{bm}_{e}})$ and original $\pi(\mu(\mathbf{x})), \mathrm{D^{l+s+c}}, \mathrm{P^{bm}})$ across multiple LLMs.
• Figure 3: Performance comparison of the fine-tuned mT5 model using only real parallel data versus incorporating varying proportions of synthetic parallel data generated by our in-context MT system. We observe a steady improvement in performance as more synthetic parallel data is added, ultimately achieving scores that match the in-context MT results of Llama3-70B.