Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming

TL;DR

The paper addresses the bottleneck of creating comprehensive test suites for automatic graders in CS1. It proposes a workflow that leverages Large Language Models (GPT-4) to generate test suites, compares them against instructor-created suites across 26 problems using over 25,000 student solutions, and analyzes mismatches to understand coverage and ambiguities. Results show LLM-generated test suites identify most valid solutions and are often as comprehensive as instructor tests, while also surfacing problem statement ambiguities that can aid instructional design; however, some invalid edge-case tests and missing edge-cases persist, necessitating human oversight. Overall, LLM-based test generation can significantly reduce instructor workload and improve autograding feedback, with practical benefits for scalable programming education and problem design improvement.

Abstract

Automatically graded programming assignments provide instant feedback to students and significantly reduce manual grading time for instructors. However, creating comprehensive suites of test cases for programming problems within automatic graders can be time-consuming and complex. The effort needed to define test suites may deter some instructors from creating additional problems or lead to inadequate test coverage, potentially resulting in misleading feedback on student solutions. Such limitations may reduce student access to the well-documented benefits of timely feedback when learning programming. In this work, we evaluate the effectiveness of using Large Language Models (LLMs), as part of a larger workflow, to automatically generate test suites for CS1-level programming problems. Each problem's statement and reference solution are provided to GPT-4 to produce a test suite that can be used by an autograder. We evaluate our proposed approach using a sample of 26 problems, and more than 25,000 attempted solutions to those problems, submitted by students in an introductory programming course. We compare the performance of the LLM-generated test suites against the instructor-created test suites for each problem. Our findings reveal that LLM-generated test suites can correctly identify most valid solutions, and for most problems are at least as comprehensive as the instructor test suites. Additionally, the LLM-generated test suites exposed ambiguities in some problem statements, underscoring their potential to improve both autograding and instructional design.

Figures (2)

• Figure 1: Detailed stages of processing a problem's solution.
• Figure 2: (a) Chaining LLM Prompts and (b) combining results from multiple LLM prompts for a function-implementation problem.