LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs

TL;DR

LANPO addresses the inefficiency of scalar rewards in RL for LLMs by separating language feedback from numerical optimization. It uses Reward-Agnostic Reflection to safely exploit intra-sample feedback and Relevant Abstraction to distill transferable lessons from inter-sample experiences, all mediated through a dynamic experience pool. The approach yields significant improvements on mathematical reasoning benchmarks for 7B and 14B models, including notable zero-shot gains and improved sample efficiency, and enables experience-driven inference without leakage. This framework offers a scalable, data-efficient path to integrate linguistic structure into RL for reasoning tasks and suggests avenues for broader applicability and automation.

Abstract

Reinforcement learning in large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts, forcing the model to explore \textit{de novo} with each attempt and hindering sample efficiency. While LLMs can uniquely learn from language feedback provided in-context, naively integrating on-line experiences into RL training presents a paradox: feedback from the same problem risks information leakage and memorization, while feedback from different problems often leads to behavior collapse due to irrelevant context. To resolve this tension, we propose \textbf{Language-And-Numerical Policy Optimization (LANPO)}, a framework that cleanly separates the roles of feedback: language guides exploration, while numerical rewards drive optimization. LANPO builds a dynamic experience pool from past trials and introduces two principles to ensure feedback is effective: \emph{Reward-Agnostic Reflection} for safe intra-sample self-correction and \emph{Relevant Abstraction} to distill generalizable lessons from inter-sample experiences. Across mathematical reasoning benchmarks, LANPO enables 7B and 14B models to significantly outperform strong baselines trained with GRPO in test accuracy. Our work provides a robust method for integrating historical experiences into the LLM RL loop, creating more effective and data-efficient learning agents.

Figures (6)

• Figure 1: Comparison between three learning paradigms for LLMs. Left: RL with numerical feedback adopts scalar rewards as the primary source of guidance for learning, where the actor cannot explicitly learn from past experiences. Middle: Test-time learning with language feedback features the LLM's ability to learn and adapt within its context window without parameter updates. Right: Our proposed language and numerical policy optimization is an RL algorithm that unifies the two by extracting meaningful language feedback from the previously discarded rollouts.
• Figure 2: Challenges in introducing language feedback to RL training. (a) At test time, both intra-sample feedback (self-correction) and inter-sample feedback (in-context examples) yield clear accuracy gains without sophisticated design. (b) However, intra-sample feedback in training suffers from information leakage: when the actor can access the ground-truth answer to the exact problem, training accuracy rises sharply but fails to translate into test-time improvement. (c) Inter-sample feedback in training, where correct rollouts are reused across problems, fails to surpass GRPO and often induces behavior collapse, in contrast to the strong ICL benefits seen in inference.
• Figure 3: Effectiveness of our algorithm design: For inter-sample feedback, we conduct similarity based selection and high-level summarization. Meanwhile, we adopt self-reflection to review intra-sample feedback and explore based on past attempts.
• Figure 4: LANPO training pipeline. The Numerical-Driven Learning Loop optimizes the policy. All attempts, whether guided by language or not, are evaluated by a Reward Model. The numerical reward is then used to update the policy via reinforcement learning. The Language-Guided Exploration Loop shapes the policy's exploration. An experience pool stores abstracted principles from past trajectories. The actor uses this pool to perform inter-sample exploration (using guidance from related problems) and intra-sample reflection (critiquing its own past attempts). Successful solutions are summarized and added back to the pool. This architecture cleanly separates the roles of feedback: language guides where to explore, while numerical rewards determine what to learn.
• Figure 5: Training dynamics. We plot key metrics for LANPO variants and the GRPO baseline on Qwen2.5-7B. (a) Mean@8 test accuracy on the AIME25 benchmark, where both feedback mechanisms improve performance throughout training process. (b) Actor gradient norm, showing stable optimization of LANPO. (c) Policy Entropy: The feedback mechanisms have distinct effects on exploration. Intra-sample feedback maintains high entropy, while inter-sample feedback reduces entropy.