Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning

TL;DR

The paper tackles the problem of multilingual reasoning being weaker than English by introducing SP3F, a two-stage, data-efficient framework that requires no target-language data. It first applies supervised fine-tuning on translated English responses to raise base correctness, then uses reinforcement learning guided by verifiable rewards and a privileged-English reference provided to a pairwise LLM judge to compare candidate outputs. By aggregating pairwise judge preferences into win-rate rewards, SP3F mitigates judge intransitivity and improves learning signals. Across $18$ languages and multiple math and non-math tasks, SP3F-7B outperforms fully post-trained baselines while using roughly $1/8$ of the post-training data, with notable gains in low-resource languages and better generalization to unseen languages. Privileged information also strengthens judge reliability and helps detect correct reasoning chains, enabling effective data-efficient multilingual reasoning.

Abstract

When asked a question in a language less seen in its training data, current reasoning large language models (RLMs) often exhibit dramatically lower performance than when asked the same question in English. In response, we introduce \texttt{SP3F} (Self-Play with Privileged Pairwise Feedback), a two-stage framework for enhancing multilingual reasoning without \textit{any} data in the target language(s). First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness. Second, we perform RL with feedback from a pairwise judge in a self-play fashion, with the judge receiving the English reference response as \textit{privileged information}. Thus, even when none of the model's responses are completely correct, the privileged pairwise judge can still tell which response is better. End-to-end, \texttt{SP3F} greatly improves base model performance, even outperforming fully post-trained models on multiple math and non-math tasks with less than of the training data across the single-language, multilingual, and generalization to unseen language settings.

Figures (7)

• Figure 1: We propose SP3F: Self-Play with Privileged Pairwise Feedback: a method for training multilingual reasoning models without any data in the target language(s). SP3F-7B out-performs Qwen2.5-7B-Instruct across 4 tasks with roughly $1/8$ of the training data ($125,000$ for SP3F-7B vs. $1,000,000$ for Qwen2.5-7B-Instruct), both in terms of accuracy and language fidelity (did the model answer in the target language?).
• Figure 2: The second stage of the SP3F pipeline is to perform RL (GRPO, shao2024deepseekmathpushinglimitsmathematical) with feedback from verifiable rewards lambert2025tulu3pushingfrontiers and a pairwise judge. To aid in its judgments, the judge LLM is given access to privileged information in the form of an English reference response. Concretely, we sample $N$ responses from the model (left), ask the privileged judge to pick a winner from each pair (center), and then use the average win-rate of each response against the other $N-1$ samples as the reward for RL (right, swamy2024minimaximalistapproachreinforcementlearning).
• Figure 3: Across 5 target languages (ordered in terms of increasing resource tier), single-language SP3F training produces models that out-performs Qwen2.5-7B-Instruct. We find particularly large deltas on lower resource languages like Indonesian, Bengali, and Swahili. Results are averaged across all four tasks considered.
• Figure 4: SP3F-7B generally outperforms Qwen2.5-7B-Instruct across most languages tested. We see particularly large gains for in-domain math tasks (left two columns) and on out-of-domain tasks in lower-resource languages (e.g., Swahili). Each bar represents the gain in terms of absolute performance points compared to Qwen2.5-7B for a specific language. N/A denotes that the task is not available for that language.
• Figure 5: As $N$ increases, it becomes increasingly likely for both privileged and non-privileged judges to have intransitive preferences. However, we consistently find that the privileged judge $\mathcal{P}_{\mathsf{priv}}$ has more transitive preferences than the non-privileged judge $\mathcal{P}_{\mathsf{no-priv}}$. We report the PNT metric proposed by xu2025investigatingnontransitivityllmasajudge. We use $N=8$ as the number of training rollouts per sample.