Evaluating Generative AI for CS1 Code Grading: Direct vs Reverse Methods

TL;DR

The paper tackles scalable, fair grading of CS1 programming assignments in large classes and the limitations of binary unit tests and manual grading. It proposes an AI-assisted grading pipeline that augments traditional evaluation by applying rubric-based scoring via LLMs and comparing two strategies, Direct and Reverse, against human TA judgments using synthetic data. It contributes an empirical comparison of Direct vs Reverse methods, examines the impact of rubric granularity (10-point vs 100-point), and emphasizes prompt design and hybrid human-AI workflows to balance efficiency with fairness. The findings suggest that, with carefully designed rubrics and prompts, AI grading can approximate TA performance, with the Reverse approach offering finer feedback on logic and potential pathways toward scalable, fair CS1 assessment.

Abstract

Manual grading of programming assignments in introductory computer science courses can be time-consuming and prone to inconsistencies. While unit testing is commonly used for automatic evaluation, it typically follows a binary pass/fail model and does not give partial marks. Recent advances in large language models (LLMs) offer the potential for automated, scalable, and more objective grading. This paper compares two AI-based grading techniques: \textit{Direct}, where the AI model applies a rubric directly to student code, and \textit{Reverse} (a newly proposed approach), where the AI first fixes errors, then deduces a grade based on the nature and number of fixes. Each method was evaluated on both the instructor's original grading scale and a tenfold expanded scale to assess the impact of range on AI grading accuracy. To assess their effectiveness, AI-assigned scores were evaluated against human tutor evaluations on a range of coding problems and error types. Initial findings suggest that while the Direct approach is faster and straightforward, the Reverse technique often provides a more fine-grained assessment by focusing on correction effort. Both methods require careful prompt engineering, particularly for allocating partial credit and handling logic errors. To further test consistency, we also used synthetic student code generated using Gemini Flash 2.0, which allowed us to evaluate AI graders on a wider range of controlled error types and difficulty levels. We discuss the strengths and limitations of each approach, practical considerations for prompt design, and future directions for hybrid human-AI grading systems that aim to improve consistency, efficiency, and fairness in CS courses.

Figures (8)

• Figure 1: Proposed AI-assisted grading pipeline for CS1 coding assignments. Submissions are first evaluated using unit tests. If they pass, grades are assigned automatically. If they fail, the code is forwarded to a large language model (LLM) for deeper evaluation using either Direct (rubric-based) or Reverse (fix-then-grade) strategies. This design enables scalable grading while supporting nuanced assessment of partially correct solutions.
• Figure 2: Example of a synthetic Moderate submission for the prime number task.
• Figure 3: Flowchart of the Direct Grading approach. The LLM receives the student's code along with a rubric, then scores each rubric category independently and outputs a rubric-aligned grade.
• Figure 4: Flowchart of the Reverse Grading approach. The LLM receives the student's code along with a rubric, but first attempts to fix the code. Based on the type and number of corrections made, it then uses the rubric to estimate a final score. This method emphasizes repair effort and debugging as part of the assessment.
• Figure 5: Box plot: Good-quality submissions across grading methods. AI (especially Reverse methods) aligns closely with TA.