Automated Feedback Generation for Undergraduate Mathematics: Development and Evaluation of an AI Teaching Assistant

TL;DR

The paper addresses the challenge of providing scalable, high-quality feedback on undergraduate mathematical proofs. It presents a modular AI teaching assistant built around large language models that processes free-form inputs, decomposes feedback tasks into configurable steps, and integrates with Lambdafeedback to deliver real-time guidance. Through new datasets and stress-testing, the authors demonstrate that, with appropriate configuration and a strong model, AI-generated feedback can approach human expert quality for early undergraduate work and reveal gaps under more challenging tasks. They also share lessons learned on prompting, data augmentation, and platform integration, offering practical guidance for deploying AI tutors in real mathematics education settings. The work has practical implications for scalability, equity, and the pedagogy of mathematical writing, while outlining future directions such as combining with formal tools and improving evaluation of feedback quality.

Abstract

Intelligent tutoring systems have long enabled automated immediate feedback on student work when it is presented in a tightly structured format and when problems are very constrained, but reliably assessing free-form mathematical reasoning remains challenging. We present a system that processes free-form natural language input, handles a wide range of edge cases, and comments competently not only on the technical correctness of submitted proofs, but also on style and presentation issues. We discuss the advantages and disadvantages of various approaches to the evaluation of such a system, and show that by the metrics we evaluate, the quality of the feedback generated is comparable to that produced by human experts when assessing early undergraduate homework. We stress-test our system with a small set of more advanced and unusual questions, and report both significant gaps and encouraging successes in that more challenging setting. Our system uses large language models in a modular workflow. The workflow configuration is human-readable and editable without programming knowledge, and allows some intermediate steps to be precomputed or injected by the instructor. A version of our tool is deployed on the Imperial mathematics homework platform Lambdafeedback. We report also on the integration of our tool into this platform.

Figures (4)

• Figure 1: High-level view of the directive workflow configured in Listing \ref{['lst:config_example']}: inputs (question, submission, generic mark scheme) feed into a small workflow graph that produces feedback, with grading as a subsequent step. In the work reported on, the grade output is only important for evaluation purposes, and not used or generated in deployment.
• Figure 2: Workflow comparison across all feedback models. Left: Median worst-case and average Pearson correlations (higher is better). The red dashed line indicates the human inter-grader correlation baseline (r=0.491). Right: Median performance range (lower indicates more consistent interpretation across graders). The number in each bar (left panel) indicates how many feedback models were tested with that workflow.
• Figure 3: Feedback length comparison across workflows. Left: Median feedback length in words for each workflow. Right: Distribution of feedback lengths showing the range and variability. The baseline_concise workflow produces the shortest feedback (median $\sim$180 words), while baseline and related workflows produce longer feedback (median $\sim$350-400 words).
• Figure 4: Performance ranges for the top 20 workflow/feedback-model combinations. Each horizontal line spans from the worst to best grading model performance, with squares indicating the average. Configurations with shorter lines demonstrate more consistent interpretation across different grading models. Red dashed lines indicate the human inter-grader correlation baseline (r=0.491 for Pearson, $\tau$=0.387 for Kendall).