MPO: Multilingual Safety Alignment via Reward Gap Optimization

TL;DR

MPO tackles multilingual safety alignment by transferring safety capabilities from a well-aligned dominant language to target languages. It does so by directly minimizing the reward-gap discrepancy between languages, using $\mathcal{L}_1 = \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} [ (\beta\,\text{RG}^t - \text{RG}^d)^2 ]$ and a dominant-language representation constraint $\mathcal{L}_2$, yielding a total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. Across three backbones (LLaMA-3.1, Gemma-2, Qwen2.5) and six target languages, MPO outperforms standard preference-learning baselines on multilingual safety benchmarks while preserving multilingual utility, particularly improving low-resource languages. Analyses show the dominant-language RG is a high-quality supervision signal and that MPO's gains are robust across data quality and quantity; visualization confirms clearer safe/unsafe boundaries in non-dominant languages after MPO. The work demonstrates that reward-gap-based supervision can scale multilingual safety without eroding overall linguistic capability, with future work extending to larger models and broader alignment tasks.

Abstract

Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility.

Figures (6)

• Figure 1: Top: Current preference learning methods optimize noisy multilingual preference data. Bottom: Our MPO directly minimizes the discrepancy of reward gap across different languages.
• Figure 2: The results of replacing the dominant language reward gap with a fixed value on multilingual safety and general utility performance.
• Figure 3: Impact of the preference data. (a) Multilingual safety performance on MultiJail with varied data quality. (b) Multilingual safety performance on MultiJail with varied data size.
• Figure 4: Reward gap across languages for the original backbone and those safety aligned by MPO and DPO.
• Figure 5: The visualization of multilingual representations for English and Swahili.