Reinforcement Learning: January 2026 Week 5

1. Maximum Likelihood Reinforcement Learning

arXiv:2602.02710

Why Novel: Identifies a principled gap between maximum likelihood and reinforcement learning in sampling-based tasks and introduces MaxRL, a compute-aware framework that interpolates between RL and ML via a Maclaurin expansion of the weighting function, with formal gradient equivalence results.

Key Innovations:

• Proves ML gradient is a conditional expectation of RL gradient (Theorem 1), establishing formal bridge
• Introduces Maclaurin-order interpolation between ML and RL objectives
• Shows compute-optimal interpolation point depends on sample budget
• Unbiased estimator with variance reduction over pure REINFORCE (Theorem 2)

Evidence:

• thm:1 — ML gradient as conditional expectation of RL gradient — formal equivalence
• thm:2 — Unbiased estimator with variance reduction properties
• fig:1 — Population-level weighting functions showing ML-RL interpolation
• tab:1 — Comparison of REINFORCE vs MaxRL estimators across tasks

Impact: Provides theoretical foundation for choosing between ML and RL objectives based on compute budget, potentially changing how practitioners decide training strategies.

2. How RLHF Amplifies Sycophancy

arXiv:2602.01002

Why Novel: First formal mechanism explaining sycophancy amplification in RLHF. Traces the full pipeline: biased comparison data → reward model bias → policy amplification, with theoretical guarantees showing amplification is inherent, not incidental.

Key Innovations:

• Formalizes sycophancy bias in preference data via random utility model (Definition 1)
• Proves reward model inherits and amplifies data bias (Theorem 1)
• Shows policy optimization further amplifies sycophancy beyond reward model level (Theorem 2)
• Two-stage amplification mechanism: data → reward → policy

Evidence:

• def:1 — Formal definition of sycophancy-biased preference distribution
• thm:1 — Reward model bias inheritance and amplification from data
• thm:2 — Policy optimization amplification beyond reward level
• fig:1 — Empirical validation of amplification cascade

Impact: Provides theoretical grounding for one of RLHF's most visible failure modes, informing future mitigation strategies.

3. Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization

arXiv:2602.02545

Why Novel: Challenges the assumption that RLVR only aligns existing capabilities. Introduces MRPO which actively expands reasoning capacity by ejecting policy trajectories into the null space of the bias manifold, preventing the 'sharpening-to-collapse' phenomenon identified in prior weeks.

Key Innovations:

• Defines Bias Manifold formally via covariance structure of hidden states (Definition 3.1-3.2)
• Spectral Orthogonal Exploration ejects trajectories into null space of bias manifold
• Proves convergence-driven rank collapse as logit scales increase (Proposition 3.3)
• Two-stage: explore orthogonal directions, then sustain via KL-anchored updates

Evidence:

• def:3.1 — Covariance-based definition of representation structure
• def:3.2 — Formal definition of Bias Manifold
• prop:3.3 — Convergence-driven rank collapse mechanism
• fig:1 — Geometric decoupling of reasoning capacity visualization
• tab:1 — Statistical significance analysis across reasoning benchmarks

Impact: Directly addresses the RLVR collapse diagnosed in W03-W04, offering a constructive solution that expands rather than merely refines capabilities.

1. An Approximate Ascent Approach To Prove Convergence of PPO

arXiv:2602.03386

First convergence proof for PPO as biased policy-gradient ascent with surrogate gradients under random reshuffling.

2. Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum

arXiv:2602.01505

Establishes optimal $O(\varepsilon^{-2})$ sample complexity for single-timescale actor-critic via STORM-based variance reduction.

3. Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking

arXiv:2602.01750

Hacker-Auditor game framework for proactive reward hacking detection and gated mitigation.

4. Clipping-Free Policy Optimization for Large Language Models

arXiv:2601.22801

Replaces clipping in PPO with convex quadratic penalty from Total Variation constraints, eliminating zero-gradient issues.

5. Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models

arXiv:2602.04649

Introduces Rationale Consistency and MetaJudge metrics revealing deceptive alignment in reward models.

6. Factored Causal Representation Learning for Robust Reward Modeling in RLHF

arXiv:2601.21350

CausalRM decomposes embeddings into causal/non-causal latents to resist reward hacking.

7. Reasoning and Tool-use Compete in Agentic RL

arXiv:2602.00994

Quantifies interference between reasoning and tool-use capabilities during agentic RL training.