Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning

TL;DR

This work introduces SPARKLE, a fine-grained framework that decomposes LLM mathematical reasoning under RL along three axes: plan-following/execution, knowledge utilization, and chain-of-subproblems. It pairs SPARKLE with SparkleRL-PSS, a two-stage RL curriculum that reuses hard problems and adds partial step scaffolding to guide exploration without new data. Empirical results show RL improves flexibility in planning and knowledge integration but struggles with robust subproblem decomposition, while human-crafted plans can hinder RL-tuned models; knowledge augmentation consistently boosts performance, especially on harder tasks. The findings offer principled guidance for designing data-efficient, interpretable RL pipelines for reasoning and underscore the value of structured auxiliary signals over static plan prescriptions.

Abstract

Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RL enhances performance is still lacking. To bridge this gap, we introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models to formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RL enhances models' ability to integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training. We introduce SparkleRL-PSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide a principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: https://sparkle-reasoning.github.io/.

Figures (16)

• Figure 1: Left: SparkleRL-PSS, a two-stage curriculum-style RL training with partial step scaffolding---the hardest problems are revisited with auxiliary step-level hints to guide multi-stage learning. Middle: Four types of auxiliary information (hints) used in this work: (1)Partial Step Scaffolding (used in Stage 2 RL), (2) High-level Plan, (3) External Knowledge, and (4) Chain of Subproblems (2-4 used in the SPARKLE analysis framework). Right: Net performance gains/losses when models are evaluated with vs. without hints (2–4). While all models struggle on compositional subproblems, RL-tuned models exhibit the greatest flexibility in leveraging auxiliary information---whereas base performance drops sharply and SFT models show limited benefit.
• Figure 2: Illustration of the SPARKLE framework's three-dimensional analysis approach. For each problem (top), we construct three complementary components: a high-level planning skeleton (left) capturing the overall solution strategy, relevant knowledge (middle) required for reasoning, and a sequence of interconnected subproblems (right) that decompose the solution process. The augmented benchmark enables a fine-grained understanding of reasoning capabilities and failure modes in LLMs.
• Figure 3: Pass@k comparison between Qwen-2.5-Math-7B, SparkleRL-Stage 1, and SparkleRL-Stage 2.
• Figure 4: Performance comparison of Qwen-2.5-Math-7B, SparkleRL-Stage 1, and SparkleRL-Stage 2 with and without additional planning information. RL-tuned models (Stage 1 and Stage 2) maintain performance with planning guidance, while the base model shows performance degradation in four out of five benchmarks when provided with plans. AIME24 exhibits the most pronounced effect where even RL-tuned models perform better without externally imposed plans.
• Figure 5: Comparison of Qwen-2.5-Math-7B, SparkleRL-Stage 1, and SparkleRL-Stage 2 with and without knowledge information. The base model shows consistent performance degradation (avg. 5.4% decrease) when provided with external knowledge, RL-tuned models show significant performance improvements (4.3% and 4.2% avg. gains for Stage 1 and Stage 2 models, respectively).