Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education

TL;DR

This work reframes LLM tutoring as an optimization problem for both visible outputs and internal reasoning by introducing PedagogicalRL-Thinking, a framework that integrates Pedagogical Reasoning Prompting grounded in Polya's four-step method with Thinking Reward to train reasoning traces. The composite reward r = $r_{\text{sol}} + (r_{\text{ped}} - 1.0) \cdot \lambda_{\text{ped}} + (r_{\text{think}} - \theta) \cdot \lambda_{\text{think}}$ guides the tutor toward higher problem-solving impact, pedagogical quality, and thinking quality, with weights $\lambda_{\text{ped}}=0.75$, $\lambda_{\text{think}}=0.3$, and threshold $\theta=0.6$. Across five ablated conditions on BigMath with synthetic student simulations, thinking-enabled tutors dramatically improve $\Delta$Solve and Helpful rates, and Ped. Think Reward achieves the best overall performance by reducing leakage while maintaining or increasing pedagogical quality. JL RL-based training also yields out-of-distribution gains on WBEB, improving pedagogical knowledge, writing scores, and decision-making while largely preserving subject knowledge, supported by a comprehensive 82-code educational codebook and both quantitative and qualitative analyses. These findings suggest that shaping internal reasoning, not just outputs, is crucial for effective AI-driven education and may generalize to other domains beyond mathematics.

Abstract

Large language models (LLMs) are increasingly deployed as intelligent tutoring systems, yet research on optimizing LLMs specifically for educational contexts remains limited. Recent works have proposed reinforcement learning approaches for training LLM tutors, but these methods focus solely on optimizing visible responses while neglecting the model's internal thinking process. We introduce PedagogicalRL-Thinking, a framework that extends pedagogical alignment to reasoning LLMs in education through two novel approaches: (1) Pedagogical Reasoning Prompting, which guides internal reasoning using domain-specific educational theory rather than generic instructions; and (2) Thinking Reward, which explicitly evaluates and reinforces the pedagogical quality of the model's reasoning traces. Our experiments reveal that domain-specific, theory-grounded prompting outperforms generic prompting, and that Thinking Reward is most effective when combined with pedagogical prompting. Furthermore, models trained only on mathematics tutoring dialogues show improved performance on educational benchmarks not seen during training, while preserving the base model's factual knowledge. Our quantitative and qualitative analyses reveal that pedagogical thinking reward produces systematic reasoning trace changes, with increased pedagogical reasoning and more structured instructional decision-making in the tutor's thinking process.

Figures (4)

• Figure 1: Overview of PedagogicalRL-Thinking. A frozen virtual student LLM interacts with an updatable tutor LLM through multi-turn dialogues. Our framework integrates two components. (1) Pedagogical Reasoning Prompting guides the tutor's internal reasoning using Polya's four-step problem-solving method. (2) Thinking Reward explicitly evaluates the pedagogical quality of reasoning traces ($r_{think}$). The composite reward function combines solve rate improvement ($r_{sol}$), pedagogical appropriateness ($r_{ped}$), and thinking quality ($r_{think}$). We evaluate tutors on $\Delta$ solve rate, leak rate, and helpfulness, and analyze reasoning traces through qualitative coding.
• Figure 2: Response characteristics across conditions. Left shows word count distribution where pedagogical conditions produce longer and more diverse responses. Center reveals mathematical content ratio with higher math engagement in thinking phases (37%) than visible responses (27-30%). Right indicates Schoenfeld phase distribution where Ped. Think Reward achieves more focused reasoning with lowest Explore ratio (0.75%). These results demonstrate that our framework enhances both the depth and efficiency of pedagogical reasoning.
• Figure 3: Codebook-based behavioral analysis. Left shows major category distribution where pedagogical conditions produce 25% mathematical problem-solving content vs. 17% for NoThink. Center presents top educational code frequencies revealing increased step-by-step instruction in pedagogical conditions. Right illustrates praise reduction from excessive (11.76% in NoThink) to appropriate levels (5.64-5.91% in pedagogical conditions). Our framework effectively reduces sycophantic behavior while promoting substantive mathematical guidance.
• Figure 4: Ablation study results. Left shows performance metrics where thinking provides the largest gain (+134% Delta Solve Rate), while thinking reward is most effective with pedagogical prompting (-19.6% Leak Rate). Right demonstrates behavioral code changes with increased mathematical problem-solving (+8.33%p) and reduced excessive praise (-5.98%p) from pedagogical prompting. Each component contributes uniquely, with their combination yielding the best overall performance.