Align to the Pivot: Dual Alignment with Self-Feedback for Multilingual Math Reasoning

TL;DR

Problem: LLMs exhibit non-uniform multilingual math reasoning, particularly in low-resource languages. Approach: PASMR designates the model’s primary language as a pivot and uses two stages—Pivot-Aligned Mapping ($Q_T$ → $Q_{ ext{En}}$, producing $A_T$) with loss $\mathcal{L}_{\text{SFT}}(\theta)$, and Self-feedback Reinforcement Learning (SRL) with token-level rewards and a PPO-style objective $L^{\text{CLIP}}(\theta)$—to enforce cross-lingual consistency without gold answers. Contributions: introduces PAM and SRL, formalizes $\mathcal{L}_{\text{SFT}}$ and $L^{\text{CLIP}}$, and demonstrates substantial multilingual gains on MGSM and MSVAMP, especially for low-resource languages, with robustness to OOD data. Significance: provides a scalable, self-supervised method for improving cross-lingual math reasoning in LLMs, reducing reliance on external translation models or gold annotations while enhancing practical multilingual problem solving.

Abstract

Despite the impressive reasoning abilities demonstrated by large language models (LLMs), empirical evidence indicates that they are not language agnostic as expected, leading to performance declines in multilingual settings, especially for low-resource languages. We attribute the decline to the model's inconsistent multilingual understanding and reasoning alignment. To address this, we present Pivot-Aligned Self-Feedback Multilingual Reasoning (PASMR), aiming to improve the alignment of multilingual math reasoning abilities in LLMs. This approach designates the model's primary language as the pivot language. During training, the model first translates questions into the pivot language to facilitate better alignment of reasoning patterns. The reasoning process in the target language is then supervised by the pivot language's reasoning answers, thereby establishing a cross-lingual self-feedback mechanism without relying on external correct answers or reward models. Extensive experimental results demonstrate that our method enhances both the model's understanding of questions and its reasoning capabilities, leading to notable task improvements.

Figures (5)

• Figure 1: Number of correct answers overlapping with English across 10 languages. Blue bars: overlapped answers; green bars: non-overlapped. (a) Mistral-7B-Instruct on MSVAMP; (b) after applying our PASMR method.
• Figure 2: During the PAM phase, the model is fine-tuned to map multilingual inputs to English and generate responses in the target language. In SRL, the model generates pivot rollouts after extracting pivot language from target language rollouts. A reward is then calculated by evaluating the consistency between the target language answer and the pivot language answer, which is then used to optimize the model.
• Figure 3: Metrics for RL training in SRL. Pivot Fail denotes the proportion of incorrect answers in the Pivot language. Pivot Correct Target Correct indicates cases where both Pivot and target answers are correct. Pivot Correct Target Fail refers to correct Pivot answers but incorrect target answers. Reward represents the average reward during training.
• Figure 4: Alignment Metrics. Answer Align / Pivot Correct — target matches pivot on correct answers; Answer Align / Pivot Fail — target matches pivot on incorrect answers.
• Figure 5: Performance of Mistral models with SRL and MSFT as training data size per language increases.