CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning
Alexandra Dragomir, Florin Brad, Radu Tudor Ionescu
TL;DR
This work tackles the data-ordering problem in multilingual MT preference learning by introducing CLewR, a curriculum learning strategy that restarts an easy-to-hard training sequence at each epoch to combat catastrophic forgetting. CLewR computes an easiness score $s$ for each translation pair using MT metrics and fixes the sample order across epochs, applying it within multiple preference optimization methods (DPO, CPO, ARPO) across several LLM families. The study demonstrates that CLewR yields consistent improvements in BLEU and COMET-22 scores, with ARPO variants using distance-based signals ($z'$) often delivering the strongest gains; CLewR-$z$ further enhances ARPO by leveraging MT metric distances. The results suggest that data ordering is a crucial lever in MT fine-tuning with PO and that the proposed restart mechanism can mitigate forgetting, providing practical gains across diverse model families; code is publicly available for replication. $s(y_w,y_l) = \frac{BLEU(y_w,y_l) + COMET(y_w,y_l) + METEOR(y_w,y_l)}{3}$, $z_\theta(y_w,y_l) = \left|\frac{\log \pi_\theta(y_w|x)}{|y_w|} - \frac{\log \pi_\theta(y_l|x)}{|y_l|}\right|$, and $z' = \eta_1 z_\theta + \eta_2 z_{BLEU} + \eta_3 z_{COMET}$ are central to the method.
Abstract
Large language models (LLMs) have demonstrated competitive performance in zero-shot multilingual machine translation (MT). Some follow-up works further improved MT performance via preference optimization, but they leave a key aspect largely underexplored: the order in which data samples are given during training. We address this topic by integrating curriculum learning into various state-of-the-art preference optimization algorithms to boost MT performance. We introduce a novel curriculum learning strategy with restarts (CLewR), which reiterates easy-to-hard curriculum multiple times during training to effectively mitigate the catastrophic forgetting of easy examples. We demonstrate consistent gains across several model families (Gemma2, Qwen2.5, Llama3.1) and preference optimization techniques. We publicly release our code at https://github.com/alexandra-dragomir/CLewR.