Beyond Final Answers: Evaluating Large Language Models for Math Tutoring

TL;DR

This work systematically evaluates LLMs for math tutoring along two axes: solving algebra problems and providing tutoring support. Using an automated tutor-based testbed with the Apprentice Tutors platform and an interactive prompting study with human evaluators, the authors quantify final-answer accuracy and tutoring quality. They find that LLMs achieve high final-answer accuracy of $85.5\%$ yet comparatively lower correctness for entire tutoring dialogues with $56.6\%$ fully correct, while the share of high-quality dialogues is about $90\%$. The results suggest LLMs can supplement math tutoring by generating problems, hints, and constructive feedback, but they are not yet reliable enough for fully autonomous tutoring without human oversight or additional reliability mechanisms.

Abstract

Researchers have made notable progress in applying Large Language Models (LLMs) to solve math problems, as demonstrated through efforts like GSM8k, ProofNet, AlphaGeometry, and MathOdyssey. This progress has sparked interest in their potential use for tutoring students in mathematics. However, the reliability of LLMs in tutoring contexts -- where correctness and instructional quality are crucial -- remains underexplored. Moreover, LLM problem-solving capabilities may not necessarily translate into effective tutoring support for students. In this work, we present two novel approaches to evaluate the correctness and quality of LLMs in math tutoring contexts. The first approach uses an intelligent tutoring system for college algebra as a testbed to assess LLM problem-solving capabilities. We generate benchmark problems using the tutor, prompt a diverse set of LLMs to solve them, and compare the solutions to those generated by the tutor. The second approach evaluates LLM as tutors rather than problem solvers. We employ human evaluators, who act as students seeking tutoring support from each LLM. We then assess the quality and correctness of the support provided by the LLMs via a qualitative coding process. We applied these methods to evaluate several ChatGPT models, including 3.5 Turbo, 4, 4o, o1-mini, and o1-preview. Our findings show that when used as problem solvers, LLMs generate correct final answers for 85.5% of the college algebra problems tested. When employed interactively as tutors, 90% of LLM dialogues show high-quality instructional support; however, many contain errors -- only 56.6% are entirely correct. We conclude that, despite their potential, LLMs are not yet suitable as intelligent tutors for math without human oversight or additional mechanisms to ensure correctness and quality.

Figures (2)

• Figure 1: Our proposed intelligent tutor-based LLM evaluation process. The process begins with the testbed setup - hosted on Google Colab (A), followed by generating problems and solutions using the intelligent tutor (Apprentice Tutors, in our case) (B). For our evaluation, 22 types of problems are then submitted to LLM models such as GPT-3.5 Turbo and GPT-4 (C). Responses from each LLM are checked by submitting them along with the tutor answers to a second LLM (D). We then performed manual human verification to validate the accuracy of the second model's responses. Finally, all results are logged into a performance tracker spreadsheet (F).
• Figure 2: Our proposed process for evaluating an LLM via interactive prompting. This diagram illustrates the process using ChatGPT's chat interface. The workflow begins by having evaluators prompt ChatGPT to provide tutor guidance on tutor problems as if they were students (A). Evaluators interactively submit queries and receive step-by-step guidance from ChatGPT (B). The evaluators systematically log each chat dialogue in a spreadsheet tracker for analysis (C).