Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

TL;DR

The paper tackles whether large language models can match expert pedagogical quality in math tutoring. It uses a controlled, turn-level comparison across expert and novice human tutors and multiple LLMs, extracting instructional moves and linguistic features to relate them to pedagogical quality. The findings show LLMs on average approach expert-level pedagogical quality, but rely on different instructional and linguistic profiles, notably underusing restating/revoicing while producing longer, more lexically diverse and more polite responses; lexical diversity and restating/revoicing strongly predict higher quality, whereas high agency and politeness predict lower quality. These results imply that improving LLM-based tutors should focus on aligning specific instructional features with pedagogical goals rather than simply increasing linguistic complexity or mimicking human tutors.

Abstract

Recent work has explored the use of large language models for generating tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We examine this question using a controlled, turn-level comparison in which expert human tutors, novice human tutors, and multiple large language models respond to the same set of math remediation conversation turns. We examine both instructional strategies and linguistic characteristics of tutoring responses, including restating and revoicing, pressing for accuracy, lexical diversity, readability, politeness, and agency. We find that large language models approach expert levels of perceived pedagogical quality on average but exhibit systematic differences in their instructional and linguistic profiles. In particular, large language models tend to underuse restating and revoicing strategies characteristic of expert human tutors, while producing longer, more lexically diverse, and more polite responses. Statistical analyses show that restating and revoicing, lexical diversity, and pressing for accuracy are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. Overall, recent large language models exhibit levels of perceived pedagogical quality comparable to expert human tutors, while relying on different instructional and linguistic strategies. These findings underscore the value of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems.

Figures (3)

• Figure 1: Instructional and linguistic profiles of tutors relative to expert tutor baseline. Each panel shows the mean difference between a tutor and the expert tutor for a given feature, with error bars indicating 95% confidence intervals. Positive values indicate higher feature values than the expert tutor, while negative values indicate lower values. The horizontal dashed line denotes parity with the expert tutor.
• Figure 2: Relative pedagogical quality across tutors. Each datapoint shows average relative pedagogical quality for each tutor (with 95% CI) when responding to the same conversation turns. The dashed line indicates parity with the turn-level average; values above (below) zero indicate higher (lower) perceived pedagogical quality relative to other tutors.
• Figure 3: Instructional and linguistic correlates of pedagogical quality. Coefficients from an ordinary least squares model predicting perceived pedagogical quality at the tutor response level. Horizontal lines indicate 95% confidence intervals. The specification includes a control for response length. Positive coefficients indicate that higher values of a feature are associated with higher perceived pedagogical quality, while negative coefficients indicate associations with lower perceived pedagogical quality. Standard errors are clustered by conversation turn to account for multiple tutor responses to the same turn.