PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning

TL;DR

PEGRL is introduced, a two-stage RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization, and a task-specific weighting scheme further balances the contributions of translation and post-editing objectives.

Abstract

Reinforcement learning (RL) has shown strong promise for LLM-based machine translation, with recent methods such as GRPO demonstrating notable gains; nevertheless, translation-oriented RL remains challenged by noisy learning signals arising from Monte Carlo return estimation, as well as a large trajectory space that favors global exploration over fine-grained local optimization. We introduce \textbf{PEGRL}, a \textit{two-stage} RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization. At each iteration, translation outputs are sampled to construct post-editing inputs, allowing return estimation in the post-editing stage to benefit from conditioning on the current translation behavior, while jointly supporting both global exploration and fine-grained local optimization. A task-specific weighting scheme further balances the contributions of translation and post-editing objectives, yielding a biased yet more sample-efficient estimator. Experiments on English$\to$Finnish, English$\to$Turkish, and English$\leftrightarrow$Chinese show consistent gains over RL baselines, and for English$\to$Turkish, performance on COMET-KIWI is comparable to advanced LLM-based systems (DeepSeek-V3.2).

Figures (8)

• Figure 1: Convergence of the GRPO group-wise baseline with respect to the number of sampled trajectories $K$. For each of 100 instances, we roll out 1024 trajectories and use the resulting baseline as a reference. We report the mean and standard deviation (error bars) of the relative gap $\Delta(K)=Q(K)-Q(1024)$, where $K$ denotes the GRPO group size. Larger $K$ reduces Monte Carlo variance (Appendix \ref{['sec:appendix_mc']}), making $Q(1024)$ a potential proxy for the true baseline $\mathbb{E}[R]$. Smaller error bars indicate more stable baseline estimation.
• Figure 2: Ablation study of our framework components on WMT24 (EN$\rightarrow$FI) and FLORES200 (EN$\rightarrow$FI), evaluated using chrF++ and COMET-KIWI. All experiments are conducted on 1K EN$\rightarrow$FI translation instances sampled from the training set. In the offline setting, an additional 7K post-editing instances are used. Models are trained for 15 epochs; at each training step, 72 trajectories are sampled per instance, and evaluation is performed every 5 steps.
• Figure 3: Gradient Weight Analysis. Experimental settings are identical to those in Section \ref{['sec:ablation']}.
• Figure 4: Convergence of the GRPO baseline estimation with respect to the number of sampled trajectories $K$ for post-editing, translation, and average translation (baseline estimation for the translation task in our framework). For each of 100 sampled instances, 1024 trajectories are rolled out and the resulting baseline is used as a reference. The figure reports the mean and standard deviation (error bars) of the relative baseline gap $\Delta(K)=Q(K)-Q(1024)$ computed from the first $K$ trajectories. Smaller error bars indicate lower variance in baseline estimation across instances, corresponding to more stable policy gradient estimates.
• Figure 5: Training dynamics on FLORES and WMT24 for EN$\rightarrow$FI under different model scales (4B, 8B), evaluated by chrF++, COMET-Kiwi, and XCOMET.

Theorems & Definitions (2)

• Theorem 1
• proof