Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When LLMs Help -- and Hurt -- Teaching Assistants in Proof-Based Courses

Romina Mahinpei, Sofiia Druchyna, Manoel Horta Ribeiro

TL;DR

A multi-part case study functioning as a technology probe in an undergraduate proof-based course finds substantial disagreement between LLMs and TAs on grading decisions but that LLM-generated feedback can still be useful to TAs for submissions with major errors.

Abstract

Teaching assistants (TAs) are essential to grading and feedback provision in proof-based courses, yet these tasks are time-intensive and difficult to scale. Although Large Language Models (LLMs) have been studied for grading and feedback, their effectiveness in proof-based courses is still unknown. Before designing LLM-based systems for this context, a necessary prerequisite is to understand whether LLMs can meaningfully assist TAs with grading and feedback. As such, we present a multi-part case study functioning as a technology probe in an undergraduate proof-based course. We compare rubric-based grading decisions made by an LLM and TAs with varying levels of expertise and examine TAs' perceptions of feedback generated by an LLM. We find substantial disagreement between LLMs and TAs on grading decisions but that LLM-generated feedback can still be useful to TAs for submissions with major errors. We conclude by discussing design implications for human-AI grading and feedback systems in proof-based courses.

When LLMs Help -- and Hurt -- Teaching Assistants in Proof-Based Courses

TL;DR

A multi-part case study functioning as a technology probe in an undergraduate proof-based course finds substantial disagreement between LLMs and TAs on grading decisions but that LLM-generated feedback can still be useful to TAs for submissions with major errors.

Abstract

Teaching assistants (TAs) are essential to grading and feedback provision in proof-based courses, yet these tasks are time-intensive and difficult to scale. Although Large Language Models (LLMs) have been studied for grading and feedback, their effectiveness in proof-based courses is still unknown. Before designing LLM-based systems for this context, a necessary prerequisite is to understand whether LLMs can meaningfully assist TAs with grading and feedback. As such, we present a multi-part case study functioning as a technology probe in an undergraduate proof-based course. We compare rubric-based grading decisions made by an LLM and TAs with varying levels of expertise and examine TAs' perceptions of feedback generated by an LLM. We find substantial disagreement between LLMs and TAs on grading decisions but that LLM-generated feedback can still be useful to TAs for submissions with major errors. We conclude by discussing design implications for human-AI grading and feedback systems in proof-based courses.
Paper Structure (18 sections, 2 figures, 4 tables)

This paper contains 18 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Study Design
  5. LLM Pipeline
  6. Results
  7. Part 1: Grading
  8. Part 2: Feedback
  9. Discussion
  10. Sample Proof Problem & Rubrics
  11. Problem Statement
  12. Solution Quality Rubric
  13. Writing Quality Rubric
  14. Prompts for LLM Pipeline
  15. Grading Prompt
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Sample survey question asking participants to rank the available feedback options for a student's submission to a subproblem.
  • Figure 2: LLM-as-a-Judge architecture for rubric-based grading and feedback provision. Two o3-mini instances independently generate candidate outputs for each subproblem while a third o3-mini instance selects the best response.