From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning

TL;DR

This work proposes an online reinforcement learning (RL)-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions by emphasizing pedagogical quality and guided problem-solving over simply giving away answers.

Abstract

Large language models (LLMs) can transform education, but their optimization for direct question-answering often undermines effective pedagogy which requires strategically withholding answers. To mitigate this, we propose an online reinforcement learning (RL)-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions by emphasizing pedagogical quality and guided problem-solving over simply giving away answers. We use our method to train a 7B parameter tutor model without human annotations which reaches similar performance to larger proprietary models like LearnLM. We introduce a controllable reward weighting to balance pedagogical support and student solving accuracy, allowing us to trace the Pareto frontier between these two objectives. Our models better preserve reasoning capabilities than single-turn SFT baselines and can optionally enhance interpretability through thinking tags that expose the model's instructional planning.

Figures (15)

• Figure 1: LLM tutoring forms a multi-objective scenario in which LLM tutors should increase the student's solve rate (y-axis) while minimizing solution leakage (x-axis). Here, the $\Delta$ solve rate measures how often a student can solve a problem before and after the dialog with a tutor and leaked solutions measures how often the tutor tells the solution to the student. Our RL-trained Qwen-2.5-7B models with varying penalty $\lambda$ are on the Pareto-front and match the performance of specialized closed-source models when tutoring on Big-Math.
• Figure 2: Workflow of our RL framework. First, we perform multiple complete student-tutor conversation simulations. After each conversation ends, the reward is computed: 1) post-dialog student solve rate (success) conditioned on the dialog, and 2) the pedagogical quality of the tutor guidance throughout the conversation. This setup uses data from the current tutor model (is on-policy) and does not use offline static dialog data (is online).
• Figure 3: Distribution of problem difficulties in our dataset (solve‑rate buckets obtained with our student model Llama‑3.1‑8B‑Instruct). The dataset contains mostly hard (1-10% solve rate) problems. This ensures each item requires meaningful guidance from the tutor model rather than being trivial for our student model.
• Figure 4: Performance of the RL tuned Qwen2.5-7B-Instruct across different $\lambda$ values: (a) student solve rate improvement, (b) leak solution rate, (c) pedagogical reward (micro).
• Figure 5: Good Example: Teacher guides the student without directly giving the answer.