Language Imbalance Driven Rewarding for Multilingual Self-improving

TL;DR

The paper introduces Language Imbalance Driven Rewarding, a self-improvement framework that treats the inherent performance gap between dominant and non-dominant languages in LLMs as a reward signal. By generating multilingual preference pairs through self-translation and optimizing with a DPO+NLL objective, the model iteratively improves across languages without human-authored data. Empirical results on Llama-3-8B-Instruct (and extensions to Qwen2-7B-Instruct) show meaningful gains in instruction-following and arithmetic reasoning across both training and unseen languages, with an observed reduction (or absence) of alignment tax on multilingual benchmarks. The approach demonstrates that leveraging language imbalance can bootstrap multilingual capabilities and generalize to new languages, offering a scalable path toward truly multilingual self-improvement of LLMs.

Abstract

Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46% win rate on the X-AlpacaEval leaderboard and 13.9% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs. The code is available at https://github.com/ZNLP/Language-Imbalance-Driven-Rewarding

Figures (4)

• Figure 1: Language Imbalance Driven Rewarding. Our method consists of two steps: (i) Self multilingual preference pair generation: Multilingual prompts are used to generate multilingual responses from $M_{t}$, respectively. Then, $M_{t}$ is utilized to perform mutual translations between responses in dominant language (e.g., en) and non-dominant languages (e.g., es, de, ru). Finally, the inherent language imbalance in LLMs is leveraged to construct multilingual preference pairs. (ii) Multilingual preference optimization: Multilingual preference pairs are constructed by $M_{t}$ itself, which are used for training via a DPO+NLL objective, resulting in model $M_{t+1}$. The whole process is iteratively repeated, enhancing the model’s multilingual abilities across all languages in each subsequent iteration, until optimization saturates.
• Figure 2: Multilingual Instruction following ability improves with Language Imbalance Driven Rewarding on Llama-3-8B-Instruct model.
• Figure 3: The reward accuracy over iterations.
• Figure 4: Multilingual Instruction following ability improves with Language Imbalance Driven Rewarding on Qwen2-7B-Instruct model.