Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mark My Works Autograder for Programming Courses

Yiding Qiu, Seyed Mahdi Azimi, Artem Lensky

TL;DR

The paper addresses the feedback bottleneck in large programming courses by introducing Mark My Works, a privacy-preserving, rubric-aware autograding system that combines unit tests with LLM-generated explanations via rubric-driven prompts. In a 2025 pilot with 191 students, AI scores showed no significant linear correlation with human scores ($r = -0.177$, $p = 0.124$), yet both methods produced left-skewed distributions; AI tended to score more conservatively while delivering richer feedback. A modular YAML-based rubric and six-module processing pipeline enabled scalable evaluation and enabled normalization to compare score distributions meaningfully. The findings suggest that AI-based feedback can complement human grading in engineering education, providing detailed guidance while preserving rubric alignment in an on-premises deployment.

Abstract

Large programming courses struggle to provide timely, detailed feedback on student code. We developed Mark My Works, a local autograding system that combines traditional unit testing with LLM-generated explanations. The system uses role-based prompts to analyze submissions, critique code quality, and generate pedagogical feedback while maintaining transparency in its reasoning process. We piloted the system in a 191-student engineering course, comparing AI-generated assessments with human grading on 79 submissions. While AI scores showed no linear correlation with human scores (r = -0.177, p = 0.124), both systems exhibited similar left-skewed distributions, suggesting they recognize comparable quality hierarchies despite different scoring philosophies. The AI system demonstrated more conservative scoring (mean: 59.95 vs 80.53 human) but generated significantly more detailed technical feedback.

Mark My Works Autograder for Programming Courses

TL;DR

The paper addresses the feedback bottleneck in large programming courses by introducing Mark My Works, a privacy-preserving, rubric-aware autograding system that combines unit tests with LLM-generated explanations via rubric-driven prompts. In a 2025 pilot with 191 students, AI scores showed no significant linear correlation with human scores (, ), yet both methods produced left-skewed distributions; AI tended to score more conservatively while delivering richer feedback. A modular YAML-based rubric and six-module processing pipeline enabled scalable evaluation and enabled normalization to compare score distributions meaningfully. The findings suggest that AI-based feedback can complement human grading in engineering education, providing detailed guidance while preserving rubric alignment in an on-premises deployment.

Abstract

Large programming courses struggle to provide timely, detailed feedback on student code. We developed Mark My Works, a local autograding system that combines traditional unit testing with LLM-generated explanations. The system uses role-based prompts to analyze submissions, critique code quality, and generate pedagogical feedback while maintaining transparency in its reasoning process. We piloted the system in a 191-student engineering course, comparing AI-generated assessments with human grading on 79 submissions. While AI scores showed no linear correlation with human scores (r = -0.177, p = 0.124), both systems exhibited similar left-skewed distributions, suggesting they recognize comparable quality hierarchies despite different scoring philosophies. The AI system demonstrated more conservative scoring (mean: 59.95 vs 80.53 human) but generated significantly more detailed technical feedback.
Paper Structure (18 sections, 1 equation, 4 figures, 3 tables)

This paper contains 18 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Mark My Works design
  5. Marking system
  6. User Interface and Backend
  7. Results
  8. Sample Methodology
  9. Raw Score Statistics
  10. Correlation Analysis
  11. Normalization and Scale Alignment
  12. Distributional Shape Comparison
  13. Discussion
  14. Assessment Approach Differences
  15. Distributional Similarity and Educational Implications
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Data Flow Pipeline from submission to final reports
  • Figure 2: System Architecture showing the modular service design
  • Figure 3: Human Score vs AI Score with Regression Analysis Correlation ($r$): -0.177, $p$-value: 0.124 Regression: Y = -0.25X + 79.71
  • Figure 4: Distribution of Human, Raw AI, and Normalized AI Scores