Maximum Likelihood Reinforcement Learning

TL;DR

This work identifies a principled gap between maximum likelihood (ML) and reinforcement learning (RL) in correctness-based tasks with non-differentiable sampling. It introduces Maximum Likelihood Reinforcement Learning (MaxRL), a compute-aware framework that interpolates between RL and ML via a Maclaurin expansion over pass@k events, with a simple on-policy gradient estimator that becomes ML in the infinite-compute limit. The authors show MaxRL Pareto-dominates standard RL baselines across diverse settings, including image classification, maze navigation, GSM8K, and large-scale LLM reasoning, delivering substantial data- and compute-efficiency advantages and better scaling with more compute and data. They also present a unifying weight-function view that clarifies how different objectives allocate learning signals across inputs, and demonstrate distinct optimization dynamics for MaxRL, notably stronger gradients on hard prompts and greater learning signal diversity. The results suggest MaxRL as a promising direction for scalable, correctness-driven RL in large models and complex tasks, with potential extensions to continuous rewards and off-policy training as future work.

Abstract

Reinforcement learning is the method of choice to train models in sampling-based setups with binary outcome feedback, such as navigation, code generation, and mathematical problem solving. In such settings, models implicitly induce a likelihood over correct rollouts. However, we observe that reinforcement learning does not maximize this likelihood, and instead optimizes only a lower-order approximation. Inspired by this observation, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a sampling-based framework to approximate maximum likelihood using reinforcement learning techniques. MaxRL addresses the challenges of non-differentiable sampling by defining a compute-indexed family of sample-based objectives that interpolate between standard reinforcement learning and exact maximum likelihood as additional sampling compute is allocated. The resulting objectives admit a simple, unbiased policy-gradient estimator and converge to maximum likelihood optimization in the infinite-compute limit. Empirically, we show that MaxRL Pareto-dominates existing methods in all models and tasks we tested, achieving up to 20x test-time scaling efficiency gains compared to its GRPO-trained counterpart. We also observe MaxRL to scale better with additional data and compute. Our results suggest MaxRL is a promising framework for scaling RL training in correctness based settings.

Figures (18)

• Figure 1: Population-level weighting functions $w(p)$.
• Figure 3: (ImageNet) Comparison of training dynamics under exact maximum likelihood, MaxRL, and REINFORCE in a controlled image classification setting. With sufficient rollouts, MaxRL closely matches cross-entropy training, while REINFORCE fails to make progress from low initial pass rates even with high number of rollouts.
• Figure 4: (Maze) Motivated by schaeffer2025largelanguagemonkeyspower, we record $-\log(\mathrm{Pass}@k)$ (lower is better) as a function of training rollouts for different objectives. We see that for across different all inference rollout budgets ($k$), MaxRL exhibit better scaling compared to GRPO and RLOO as we increase number of training rollouts.
• Figure 5: (GMS8K) Training dynamics on GSM8K with a fixed dataset and increasing training compute in terms of RL steps. MaxRL shows slower initial gains but ultimately achieves higher performance with substantially less pass@k degradation compared to GRPO and REINFORCE.
• Figure 6: (Qwen3 training results) Evaluation of final checkpoints from training Qwen3-1.7B-Base and Qwen3-4B-Base models, on 4 benchmarks: AIME 2025, BeyondAIME, MATH-500 and Minerva. MaxRL match or ourperform GRPO in all 4 evaluation datasets and shows little to no degradation at coverage (pass@k) for very high k values. We also note the increase in inference efficiency: MaxRL can provide $2.3 \times$ - $19.2\times$ speedup compared to GRPO while generating multiple samples with a perfect verifier and maintains similar or better pass@1 performance.

Theorems & Definitions (8)

• theorem 1: Conditional Form of the Maximum Likelihood Gradient
• theorem 2: Estimator--objective equivalence
• theorem 3: Restatement of Theorem \ref{['prop:ml-conditional']}
• proof
• theorem 4: Restatement of Theorem \ref{['prop:logT_equivalence']}
• proof
• proposition 1
• proof