MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors

TL;DR

MathTutorBench tackles the lack of holistic, scalable evaluation for open-ended math tutoring by LLMs. It defines three skill domains—subject expertise, student understanding, and pedagogical abilities—and curates datasets (GSM8k, Bridge, MathDial) into MathDialBridge variants, plus a StepVerify, to enable comprehensive automatic evaluation. A reward-model-based Scaffolding Score and a pairwise preference data pipeline are used to assess pedagogical quality across seven tutoring tasks, revealing a trade-off between solving ability and pedagogy and greater difficulty in longer dialogs. The benchmark is open-source to accelerate rapid, fair comparisons and the development of safer, more effective tutoring LLMs, while acknowledging limitations and the need for extension to broader domains and safety considerations.

Abstract

Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.

Figures (7)

• Figure 1: Effective teaching requires various skills which we categorize into expertise, student understanding, and pedagogical ability. MathTutorBench evalutes these according to the tasks shown in the outer ring.
• Figure 2: Overview of the MathTutorBench benchmark. Each benchmark task defines a dataset, system prompts with problem and dialog, metric, and ground-truth teacher responses. A reward model is used to score the pedagogical quality over teacher responses (win rate). The right part of the figure shows the outcome as a performance comparison of selected LLMs. While they all perform well in a simple problem-solving setting, most of them lack in correct detection of mistakes and generating pedagogical responses.
• Figure 3: Reward model distribution scores for expert and novice teachers across prompted (prompt in Figure \ref{['fig:simple-prompt-rm']}), with extended prompt (prompt in Figure \ref{['fig:prompt-reward-model']}), and finetuned Qwen2.5-1.5B-Instruct models.
• Figure 4: Models performance on pairwise judgment of teacher responses. We compute accuracy on an independent test set based on Bridge dataset bridge24 as a proportion of expert teacher responses preferred over novice teacher responses. Extended prompt enumerates our pedagogical criteria (Figure \ref{['fig:prompt-reward-model']}).
• Figure 5: Exact prompts for each task.