Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning

TL;DR

This work tackles the limited informational feedback in reinforcement learning with verifiable rewards (RLVR) by introducing Adaptive Ability Decomposing (A^2D). A^2D first trains a decomposer to break complex questions into simpler sub-questions, then annotates training data with these sub-questions and trains a reasoner under RLVR with sub-question guidance via an in-context distillation loss (IDL). The approach is plug-and-play, yielding consistent improvements across multiple RLVR algorithms and backbone models on eight mathematical benchmarks, and it provides a detailed analysis of how decomposer training and hint granularity affect exploration and exploitation. The findings suggest coarse-grained sub-question guidance enhances exploration while fine-grained prompts improve exploitation, offering practical pathways to scale large reasoning systems without external teacher models.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.

Figures (4)

• Figure 1: Average performance of various models on mathematical tasks. By incorporating sub-question guidance into the prompts, LLMs can perform exploration and reasoning more effectively, thereby achieving higher Pass@k scores.
• Figure 2: The comparison between vanilla RLVR and our approach A$^2$D. In A$^2$D, we first train a decomposer $\pi_{\theta_\text{D}}$ through RLVR, guided by the format reward $R_\text{F}$ and quality reward $R_\text{Q}$. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and design the in-context distillation loss (IDL), i.e., Eq. \ref{['eq:idl']}, to integrate the sub-question guidance into the RLVR process for reasoner $\pi_{\theta_\text{R}}$. To better demonstrate the pipeline of our A$^2$D, we present the pseudo code in Algorithm \ref{['code:a2d']}.
• Figure 3: Performance comparison of Qwen2.5-7B-Instruct of whether sub-questions are included in the prompt.
• Figure 4: An example of the generated sub-question from models trained through different methods.