The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities

TL;DR

The work addresses whether fine-tuning LLMs on parallel translation data preserves or erodes LLM-specific advantages such as steerability, document-level translation, and non-literal rendering. By evaluating LLaMA and Falcon models across 7B–65B scales and six language directions, the authors quantify both general translation gains (via COMET) and qualitative behaviors, revealing that parallel-data fine-tuning generally improves translation quality but degrades formality steering, document-contextualization, and technical-domain few-shot translation. Introducing a mixed fine-tuning regimen that combines monolingual and parallel data mitigates these degradations while delivering larger overall translation gains, demonstrating a practical path to preserve LLM capabilities in MT. The IdiomsInCtx-MT evaluation further emphasizes the potential for improved non-literal translation, and the results collectively call for MT-specific fine-tuning strategies that preserve qualitative LLM strengths while boosting translation quality, with implications for deployment across diverse domains and languages.

Abstract

Fine-tuning large language models (LLMs) for machine translation has shown improvements in overall translation quality. However, it is unclear what is the impact of fine-tuning on desirable LLM behaviors that are not present in neural machine translation models, such as steerability, inherent document-level translation abilities, and the ability to produce less literal translations. We perform an extensive translation evaluation on the LLaMA and Falcon family of models with model size ranging from 7 billion up to 65 billion parameters. Our results show that while fine-tuning improves the general translation quality of LLMs, several abilities degrade. In particular, we observe a decline in the ability to perform formality steering, to produce technical translations through few-shot examples, and to perform document-level translation. On the other hand, we observe that the model produces less literal translations after fine-tuning on parallel data. We show that by including monolingual data as part of the fine-tuning data we can maintain the abilities while simultaneously enhancing overall translation quality. Our findings emphasize the need for fine-tuning strategies that preserve the benefits of LLMs for machine translation.

Figures (11)

• Figure 1: X$\rightarrow$English (top) and English$\rightarrow$X (bottom) COMET scores on WMT22 for models trained on human-written translations with different amounts of training data.
• Figure 2: Accuracy of formality markers for models trained on human-written translations.
• Figure 3: COMET on technical domains using 5-shot examples for models trained on human-written translations.
• Figure 4: Accuracy of animacy contextualization for German$\rightarrow$English and Russian$\rightarrow$English for models fine-tuned on human-written translations.
• Figure 5: COMET, LitTER and MWEScore on IdiomsInCtx-MT test sets for LLaMA-65b fine-tuned on human-quality parallel data.