ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning

TL;DR

This work addresses improving machine translation for large reasoning models (LRMs) by introducing ExTrans, an exemplar-enhanced reinforcement learning framework that leverages LLM-as-exemplar and LLM-as-judge signals to shape MT rewards. The core method defines a composite reward $r_{all}$ from format, thought, exemplar-translation, and auxiliary metrics, and extends this approach to multilingual MT via a lightweight generalization reward, enabling transfer across 11 languages. Empirical results show ExTrans-7B achieves state-of-the-art performance on English→Chinese literary translation and that the multilingual variant mExTrans-7B delivers competitive performance across 90 translation directions, with ablations confirming the importance of exemplar-guided translation and CometKiwi signals. The approach offers a practical path to scalable, high-quality MT for LRMs in both single-language and multilingual settings, albeit with costs associated with exemplar generation and remaining gaps relative to the strongest baselines in certain low-resource directions. Overall, the combination of exemplar-based rewards and lightweight multilingual generalization demonstrates a significant advance in aligning LM reasoning capabilities with translation quality for literary and cross-language MT tasks.

Abstract

In recent years, the emergence of large reasoning models (LRMs), such as OpenAI-o1 and DeepSeek-R1, has shown impressive capabilities in complex problems, e.g., mathematics and coding. Some pioneering studies attempt to bring the success of LRMs in neural machine translation (MT). They try to build LRMs with deep reasoning MT ability via reinforcement learning (RL). Despite some progress that has been made, these attempts generally focus on several high-resource languages, e.g., English and Chinese, leaving the performance on other languages unclear. Besides, the reward modeling methods in previous work do not fully unleash the potential of reinforcement learning in MT. In this work, we first design a new reward modeling method that compares the translation results of the policy MT model with a strong LRM (i.e., DeepSeek-R1-671B), and quantifies the comparisons to provide rewards. Experimental results demonstrate the superiority of the reward modeling method. Using Qwen2.5-7B-Instruct as the backbone, the trained model achieves the new state-of-the-art performance in literary translation, and outperforms strong LRMs including OpenAI-o1 and DeepSeeK-R1. Furthermore, we extend our method to the multilingual settings with 11 languages. With a carefully designed lightweight reward modeling in RL, we can simply transfer the strong MT ability from a single direction into multiple (i.e., 90) translation directions and achieve impressive multilingual MT performance.

Figures (2)

• Figure 1: The illustration of the exemplar-enhanced translation reward.
• Figure 2: Performance and Rewards change of ExTrans-7B during RL training. Among them, (a) and (b) are conducted on the MetaphorTrans test set, while (c) and (d) are conducted on its training set. The horizontal axis denotes the number of training steps, and there are 600 steps (2 epochs) in total. The vertical axis denotes the value of the corresponding metrics.