Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization

Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Yang Li

TL;DR

This work challenges the notion that scaling alone governs reasoning capacity by introducing Manifold-Reshaping Policy Optimization (MRPO), a two-stage framework that first ejects policy trajectories into the null space of the bias manifold via Spectral Orthogonal Exploration and then sustains high-dimensional reasoning with an Effective Rank regularization in Rank-Aware GRPO. Empirically, a 4B-parameter model trained with MRPO surpasses larger models on challenging math benchmarks, demonstrating improved pass@1 and sustained high-rank trajectories with favorable token efficiency. The paper provides a geometric interpretation of alignment, defines the Bias Manifold and Reasoning Collapse, and presents a practical pathway to a Geometric Scaling Law that could reduce reliance on parameter growth while enhancing reasoning capabilities. These contributions have implications for scalable, efficient, and potentially safer AI systems, though they acknowledge complexity and safety considerations when operating outside traditional alignment boundaries.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold. In this work, we challenge this accessibility boundary hypothesis by demonstrating that the latent reasoning space can be fundamentally expanded through targeted geometric interventions. We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs. MRPO operates in two stages: first, we employ Spectral Orthogonal Exploration (SOE) to eject the policy initialization into the null space of the bias manifold; second, we integrate an Effective Rank regularization term into the policy optimization objective. This approach incentivizes the discovery and maintenance of high-dimensional reasoning trajectories against the entropy-reducing tendency of standard RL. Empirically, our 4B-parameter method achieves state-of-the-art performance on mathematical tasks, significantly outperforming larger models (e.g., Qwen3-32B) and expanding the capability boundary beyond standard GRPO. Our code is available at https://anonymous.4open.science/r/MRPO-D57B/

Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization

TL;DR

This work challenges the notion that scaling alone governs reasoning capacity by introducing Manifold-Reshaping Policy Optimization (MRPO), a two-stage framework that first ejects policy trajectories into the null space of the bias manifold via Spectral Orthogonal Exploration and then sustains high-dimensional reasoning with an Effective Rank regularization in Rank-Aware GRPO. Empirically, a 4B-parameter model trained with MRPO surpasses larger models on challenging math benchmarks, demonstrating improved pass@1 and sustained high-rank trajectories with favorable token efficiency. The paper provides a geometric interpretation of alignment, defines the Bias Manifold and Reasoning Collapse, and presents a practical pathway to a Geometric Scaling Law that could reduce reliance on parameter growth while enhancing reasoning capabilities. These contributions have implications for scalable, efficient, and potentially safer AI systems, though they acknowledge complexity and safety considerations when operating outside traditional alignment boundaries.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold. In this work, we challenge this accessibility boundary hypothesis by demonstrating that the latent reasoning space can be fundamentally expanded through targeted geometric interventions. We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs. MRPO operates in two stages: first, we employ Spectral Orthogonal Exploration (SOE) to eject the policy initialization into the null space of the bias manifold; second, we integrate an Effective Rank regularization term into the policy optimization objective. This approach incentivizes the discovery and maintenance of high-dimensional reasoning trajectories against the entropy-reducing tendency of standard RL. Empirically, our 4B-parameter method achieves state-of-the-art performance on mathematical tasks, significantly outperforming larger models (e.g., Qwen3-32B) and expanding the capability boundary beyond standard GRPO. Our code is available at https://anonymous.4open.science/r/MRPO-D57B/
Paper Structure (31 sections, 1 theorem, 10 equations, 7 figures, 6 tables)

This paper contains 31 sections, 1 theorem, 10 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries: Formalizing the Trap
  4. The Geometry of Latent Representations
  5. Decoupling Geometry from Uncertainty
  6. The Bias Manifold Hypothesis
  7. Reasoning Collapse and Geometric Barriers
  8. Methodology: Manifold-Reshaping Policy Optimization (MRPO)
  9. Stage I: Geometric Ejection via Spectral Orthogonal Exploration (SOE)
  10. Stage II: Rank-Aware Group Relative Policy Optimization
  11. Effective Rank Reward
  12. Rank-Aware GRPO Objective
  13. Experiments
  14. Experimental Setup
  15. Main Results
  16. ...and 16 more sections

Key Result

Proposition 3.3

As the policy $\pi_\theta$ converges, increasing logit scales reduce the effective sampling temperature. This acts as a rank-reducing operator, causing the effective rank of generated trajectories to contract: trapping the model in a low-dimensional bias manifold.

Figures (7)

  • Figure 1: Geometric Decoupling of Reasoning Capacity.
  • Figure 2: Geometric Interpretation of Our Method's Framework.
  • Figure 3: Visualization of performance comparison. MRPO demonstrates superior accuracy on AIME and MATH benchmarks compared to baselines.
  • Figure 4: Comparison of Rank.
  • Figure 5: Unbiased Pass@K Analysis on AIME Benchmarks. We evaluate the reasoning coverage of MRPO (Red) versus Pure GRPO (Grey) on AIME 2024 and AIME 2025.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Definition 3.1
  • Definition 3.2: Local Bias Manifold
  • Proposition 3.3: Confidence-Induced Rank Collapse