Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation

TL;DR

RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then played to prevent forgetting and presents a pathway for creating more versatile LLMs that excel in specialized tasks without compromising generality and safety.

Abstract

Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks but still encounter difficulties in machine translation. Traditional methods to improve translation have typically involved fine-tuning LLMs using parallel corpora. However, vanilla fine-tuning often leads to catastrophic forgetting of the instruction-following capabilities and alignment with human preferences, compromising their broad general abilities and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make existing continual instruction tuning methods ineffective. To overcome this issue, we propose a novel approach called RaDis (Rationale Distillation). RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then "replayed" to prevent forgetting. These rationales encapsulate general knowledge and safety principles, acting as self-distillation targets to regulate the training process. By jointly training on both reference translations and self-generated rationales, the model can learn new translation skills while preserving its overall general abilities. Extensive experiments demonstrate that our method enhances machine translation performance while maintaining the broader capabilities of LLMs across other tasks. This work presents a pathway for creating more versatile LLMs that excel in specialized tasks without compromising generality and safety.

Figures (13)

• Figure 1: Translation performance (COMET) and general conversational and instruction-following ability (MT-Bench). While both Fine-tuning (blue triangle) and RaDis (red star) greatly enhance the translation performance, RaDis helps preserve most of the models' general ability.
• Figure 2: An example of LLM's response to translation instruction. In this case, the LLM provides a rationale with additional factual information about the term 'Belt and Road' after the translation result.
• Figure 3: Overview of the RaDis approach. Rationale Generating (Left): Given a translation instruction-response pair as an input, the LLM extends the response by generating a rationale. Fine-tuning with Rationale Distillation (Right): RaDis utilizes this self-generated rationale to enrich the original response and fine-tunes the LLM with the enriched data. The CLM loss computed on the rationale serves as a self-distillation regularization term, preventing excessive parameter divergence.
• Figure 4: Overview of the gradient similarity between the regularization term and MT loss.
• Figure 5: The translation instructions.