Beyond Correctness: Learning Robust Reasoning via Transfer

TL;DR

The paper tackles robustness of LLM reasoning beyond attaining the correct final answer by introducing RLTR, a reinforcement learning framework that adds a transfer reward. This reward measures whether a receiver model can continue a truncated reasoning prefix from a generator to reach the correct answer, promoting reasoning that is stable and reusable across models. Empirical results across math and science benchmarks show that RLTR improves both average accuracy and multi-sample consistency (Maj@K), while achieving faster convergence and lower overall compute than RLVR. Analyses reveal a strong link between transferability and robustness, with ablations demonstrating the importance of the transfer weight and receiver capacity. The approach offers a scalable, domain-agnostic signal for shaping robust reasoning in RL-based alignment and optimization of LLMs.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the reasoning process itself. We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it, and treat reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable reasoning and significantly more sample efficient.

Figures (4)

• Figure 1: Overview of RLTR: RLTR augments standard RLVR with a transfer reward. Top: A trainable generator model produces a full completion, whose final-answer correctness yields on answer reward for policy optimization. Bottom: We then truncate the generated reasoning to form a prefix and feed it to a frozen receiver model to produce continued completions whose final-answer correctness defines a transfer reward that measures the transferability of partial reasoning across models. The answer reward and transfer reward are combined into a unified reward signal used to update the generator policy.
• Figure 2: Training dynamics and transferability.(a)Average accuracy: Our method RLTR, significantly reduce the training steps to achieve comparable performance with RLVR-trained models. Notably, RLTR matches RLVR with $2.5\times$ fewer training steps. (b)Maj@64: Compared to RLVR, RLTR progressively improves consistency over training, whereas RLVR’s performance degrades as training proceeds. (c)Transferability: Transferability follows Maj@64, supporting our view that optimizing the transfer reward through RLTR improves reasoning consistency and robustness. Across all three views, RLTR consistently outperforms RLVR, indicating a shift toward reliably correct dominant solution modes and more stable optimization progress under the proposed transfer-based reward.
• Figure 3: RLTR improves transferability consistently. Transferability (%) comparison among base model, RLVR, and RLTR. The truncation ratio is the fraction of tokens remained from the begging of a reasoning trace (higher means longer prefixes continuation). RLTR consistently achieves higher transferability than baselines.
• Figure 4: Detailed Analysis of transferability on various truncation ratios $\tau\in\{0.3,0.5,0.7,0.9\}$.