Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models

TL;DR

This paper systematically reexamines the six classical MT challenges in the context of Large Language Models, using Llama2-7B/13B with SFT and CPT-SFT training paradigms to translate German–English and English-to-X. It finds that LLMs reduce reliance on bilingual parallel data for major languages and substantially improve long-sentence and document-level translation, yet domain mismatch and rare-word prediction persist, especially for low-resource languages. Scaling up to larger LLMs provides gains but introduces data-volume sensitivity and ongoing rare-word weaknesses, while new challenges such as inference efficiency and robust human-aligned evaluation emerge. The work highlights the importance of balanced pretraining data, efficient inference techniques, and evaluation methods that align with human judgment, and it provides public release of datasets and models to spur further progress.

Abstract

The evolution of Neural Machine Translation (NMT) has been significantly influenced by six core challenges (Koehn and Knowles, 2017), which have acted as benchmarks for progress in this field. This study revisits these challenges, offering insights into their ongoing relevance in the context of advanced Large Language Models (LLMs): domain mismatch, amount of parallel data, rare word prediction, translation of long sentences, attention model as word alignment, and sub-optimal beam search. Our empirical findings indicate that LLMs effectively lessen the reliance on parallel data for major languages in the pretraining phase. Additionally, the LLM-based translation system significantly enhances the translation of long sentences that contain approximately 80 words and shows the capability to translate documents of up to 512 words. However, despite these significant improvements, the challenges of domain mismatch and prediction of rare words persist. While the challenges of word alignment and beam search, specifically associated with NMT, may not apply to LLMs, we identify three new challenges for LLMs in translation tasks: inference efficiency, translation of low-resource languages in the pretraining phase, and human-aligned evaluation. The datasets and models are released at https://github.com/pangjh3/LLM4MT.

Figures (9)

• Figure 1: BLEU and COMET-DA scores for German-to-English systems, with "0" on the x-axis indicating models trained exclusively on the Alpaca dataset. LLMs reduce reliance on extensive parallel data.
• Figure 2: BLEU and COMET-DA scores for German-to-English systems. Increased parallel data adversely affects the performance of the Llama2-13B model.
• Figure 3: Precision of translation and delete rates by source word frequency. The light blue and dark green indicate the best LLM-SFT and Enc2Dec translation models. The horizontal axis represents the frequency of source word types in the test corpus, where axis labels indicate the upper limit of each frequency range, and the bin width is proportional to the number of word types in that range. The precision and deletion rates are shown on the upper and lower vertical axes respectively. LLMs excel at predicting words that appear more than eight times but perform poorly with rare words.
• Figure 4: BLEU scores for German-to-English MT systems with varying sample lengths. Sentence-level translation involves lengths below $90$ words, while document-level translation concerns longer samples. LLMs improve long-sentence translation and consistently excel in document-level tasks.
• Figure 5: COMET-DA scores for sentence-level translation of German-to-English MT systems with varying sample lengths.