Decomposing Elements of Problem Solving: What "Math" Does RL Teach?

TL;DR

This work investigates how reinforcement learning (specifically GRPO) affects mathematical reasoning in large language models by decomposing problem solving into Plan, Execute, and Verify. It shows that GRPO primarily boosts execution robustness and reduces sensitivity to sampling temperature (a phenomenon termed temperature distillation), without expanding the set of solvable problems (coverage). A minimal synthetic task is constructed to replicate these effects and to identify conditions under which RL could overcome the coverage wall, notably through exploration and generalization to new solution paths. The findings provide a nuanced view of RL’s role in reasoning, revealing both its limitations and a potential path to enhance problem-solving coverage via controlled exploration and data design.

Abstract

Mathematical reasoning tasks have become prominent benchmarks for assessing the reasoning capabilities of LLMs, especially with reinforcement learning (RL) methods such as GRPO showing significant performance gains. However, accuracy metrics alone do not support fine-grained assessment of capabilities and fail to reveal which problem-solving skills have been internalized. To better understand these capabilities, we propose to decompose problem solving into fundamental capabilities: Plan (mapping questions to sequences of steps), Execute (correctly performing solution steps), and Verify (identifying the correctness of a solution). Empirically, we find that GRPO mainly enhances the execution skill-improving execution robustness on problems the model already knows how to solve-a phenomenon we call temperature distillation. More importantly, we show that RL-trained models struggle with fundamentally new problems, hitting a 'coverage wall' due to insufficient planning skills. To explore RL's impact more deeply, we construct a minimal, synthetic solution-tree navigation task as an analogy for mathematical problem-solving. This controlled setup replicates our empirical findings, confirming RL primarily boosts execution robustness. Importantly, in this setting, we identify conditions under which RL can potentially overcome the coverage wall through improved exploration and generalization to new solution paths. Our findings provide insights into the role of RL in enhancing LLM reasoning, expose key limitations, and suggest a path toward overcoming these barriers. Code is available at https://github.com/cfpark00/RL-Wall.

Figures (34)

• Figure 1: Decomposing problem solving and modeling the effects of RL in a toy setting.(a): We decompose math problem solving into fundamental capabilities. To solve a problem model must first plan (map the question to a sequence of actions) and then execute (navigate the tree) without errors. (b): We design a minimal toy environment that explicitly captures these core reasoning elements in a controlled setting. (c): GRPO primarily improves precision on problems the model already partially solves, without expanding overall coverage. Our synthetic setup reproduces this behavior—and crucially, shows that under the right conditions, RL can break the coverage wall.
• Figure 2: GRPO improves precision (average@$K$) through temperature distillation but does not improve coverage (measured as pass@$K$).Left, Middle: Precision (probability of correctly solving the problem) for MATH-500 questions before and after performing GRPO on Qwen2.5-1.5B-Instruct. GRPO "tightens" the model's behaviors, which we study as the "temperature distillation effect". Right: Pass@64 at various temperature settings (solid lines). We define coverage as the best Pass@64 rate over temperatures (dashed lines). GRPO does not improve coverage.
• Figure 3: Per-problem precision improvement on train subset (left) and test set (right). We sort problems by pre-GRPO model's precision, and examine the improvement from running GRPO. We report mean precision and standard deviations from 5 random seeds. GRPO improves precision on questions of medium-level difficulty while encountering a train-test generalization gap (green, purple star). GRPO also encounters a coverage wall (yellow) on hard problems.
• Figure 4: Two elements to ace a math exam: Plan then Execute.Left: Examples of planning and execution failures from Qwen2.5-1.5B-Instruct on MATH-500 questions. Right: For MATH-500, even a small model knows how to approach (i.e., plan) most problems. However, the ability to correctly executing them depends heavily on model sizes. GRPO has a larger impact on a model's execution ability (i.e., improving execution robustness).
• Figure 5: Model cannot reliably execute math and logic reasoning due to spurious correlations.Left: One example of solution tree from Qwen2.5-1.5B-Instruct on a MATH-500 question. In this question, the model's likelihood of outputting the correct solution rely heavily on a semantically irrelevant context (saying "unit" versus "block"). Right: To examinbe whether GRPO fixes spurious correlations, we annotate all the generations with execution failures (Figure \ref{['fig:human_annot']}), and examine the relative number of mistakes before and after GRPO: GRPO helps model fixes basic math and logic errors but does not improve advanced (high school) math.