Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

TL;DR

The paper tackles the problem of optimizing large language model policies by revealing that the policy secretly comprises internal Layer and Modular policies that evolve across Transformer residual streams. It introduces a formal internal-policy decomposition, analyzes entropy dynamics to identify progressive reasoning patterns—especially a three-stage Exploration-Integration-Convergence in Qwen and abrupt final-layer convergence in Llama—and demonstrates that bottom-up optimization (BuPO) of internal policies yields superior reasoning performance on complex benchmarks. The authors extend GRPO with InterGRPO for internal-layer optimization and present BuPO, which aligns low-layer policies early to reconstruct foundational reasoning, yielding consistent gains across multiple model families. This layer-aware RL paradigm provides practical guidance for more efficient and robust reasoning in LLMs and opens avenues for architecture-aware optimization and interpretability-driven RL design.

Abstract

Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a single unified policy, overlooking their internal mechanisms. Understanding how policy evolves across layers and modules is therefore crucial for enabling more targeted optimization and raveling out complex reasoning mechanisms. In this paper, we decompose the language model policy by leveraging the intrinsic split of the Transformer residual stream and the equivalence between the composition of hidden states with the unembedding matrix and the resulting samplable policy. This decomposition reveals Internal Layer Policies, corresponding to contributions from individual layers, and Internal Modular Policies, which align with the self-attention and feed-forward network (FFN) components within each layer. By analyzing the entropy of internal policy, we find that: (a) Early layers keep high entropy for exploration, top layers converge to near-zero entropy for refinement, with convergence patterns varying across model series. (b) LLama's prediction space rapidly converges in the final layer, whereas Qwen-series models, especially Qwen3, exhibit a more human-like, progressively structured reasoning pattern. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that directly optimizes the internal layer policy during early training. By aligning training objective at lower layer, BuPO reconstructs foundational reasoning capabilities and achieves superior performance. Extensive experiments on complex reasoning benchmarks demonstrates the effectiveness of our method. Our code is available at https://github.com/Trae1ounG/BuPO.

Figures (11)

• Figure 1: (a): The residual stream within Transformer which moves from previous layer hidden states into self-attention and feed-forward network (FFN) sequentially. (b): Any hidden states with unembedding matrix $\mathbf{E}_\text{u}$ can be transformed into probability distribution $\mathbf{P}$ over the vocabulary space, which can be considered as the samplable policy. (c): We surprisingly find that Qwen-series contains a progressive reasoning pattern in FFN, where start from exploration expansion to integrate middle layer knowledge into final prediction convergence, specially in Qwen3.
• Figure 2: Continuous entropy dynamics of internal policy for different models. The residual stream flows from $\mathbf{H}^{l-1}$ into $\mathbf{A}^l$, $\mathbf{F}^l$, and finally to the next layer $\mathbf{H}^l$.
• Figure 3: Entropy Change Dynamics of Internal Policy. The entropy change $\Delta H^l$ across layers represents the uncertainty of current policy's hidden exploration space. A positive $\Delta H^l > 0$ indicates increasing exploration, $\Delta H^l \approx 0$ signifies exploitation of existing knowledge, and $\Delta H^l < 0$ suggests a tendency toward convergence within the reasoning process.
• Figure 4: Residual cosine similarity across different Qwen models
• Figure 5: Training dynamics of internal policy. Effects of varying the optimized policy on (a) reward, (b) entropy of language model policy $\pi_\theta$, (c) response length. The backbone model is Qwen3-4B.