Simulated Students in Tutoring Dialogues: Substance or Illusion?

TL;DR

This work tackles the challenge of evaluating LLM-based tutoring systems using simulated students by proposing a reference-based, turn-level evaluation framework grounded in six behavioral dimensions. It benchmarks prompting, fine-tuning, and offline RL approaches on a real-world 2k-question math tutoring dataset, revealing that prompting alone often fails to capture key dimensions of student behavior while supervised fine-tuning and reward-based optimization yield substantial, though still imperfect, improvements. The study validates metrics through human evaluation and shows strong agreement between automated and expert judgments, highlighting both the promise and the current limitations of simulated students for training and evaluating AI tutors. The findings motivate future work on more advanced RL techniques, richer contextualization, and domain-general evaluation metrics to better approximate authentic student learning trajectories and dialogue dynamics.

Abstract

Advances in large language models (LLMs) enable many new innovations in education. However, evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. Therefore, many recent works on LLM-powered tutoring solutions have used simulated students for both training and evaluation, often via simple prompting. Surprisingly, little work has been done to ensure or even measure the quality of simulated students. In this work, we formally define the student simulation task, propose a set of evaluation metrics that span linguistic, behavioral, and cognitive aspects, and benchmark a wide range of student simulation methods on these metrics. We experiment on a real-world math tutoring dialogue dataset, where both automated and human evaluation results show that prompting strategies for student simulation perform poorly; supervised fine-tuning and preference optimization yield much better but still limited performance, motivating future work on this challenging task.

Figures (6)

• Figure 1: Overview of the seven evaluation metrics for simulated student turn evaluation, with a real paraphrased tutor–student dialogue serving as the reference. In this example, the ground-truth student turn and the simulated turn have the same dialogue act Math Answer; however, the target turn is incorrect while the simulated turn is correct.
• Figure 2: Distribution of act labels for real students and simulated student methods.
• Figure 3: Distribution of correctness labels for real students and simulated student methods.
• Figure 4: Results across metrics and methods broken down for each turn pair index.
• Figure 5: Human evaluation interface for evaluating ground-truth turns.