Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond human subjectivity and error: a novel AI grading system

Alexandra Gobrecht, Felix Tuma, Moritz Möller, Thomas Zöller, Mark Zakhvatkin, Alexandra Wuttig, Holger Sommerfeldt, Sven Schütt

TL;DR

The paper presents ASAG, a novel automatic short-answer grading system built on a fine-tuned open-source transformer trained on a large, cross-disciplinary university exam dataset. It demonstrates strong generalization to unseen questions and courses, achieving MAE around 1.32–1.44 points, RMSE around 2.27–2.41, and Pearson correlations near 0.69–0.78 on held-out sets, and outperforms human re-graders in matching official grades in a large-scale human benchmark. The study also outlines a concrete architecture, a four-element input/output formulation, and a rigorous evaluation framework including normalization, diverse splits, and a detailed human-vs-AI comparison, underscoring the potential of AI-assisted or autonomous grading. Practical implications include safer deployment via assisted grading levels, explainability enhancements, and a roadmap toward fairer, more consistent grading while reducing educator workload.

Abstract

The grading of open-ended questions is a high-effort, high-impact task in education. Automating this task promises a significant reduction in workload for education professionals, as well as more consistent grading outcomes for students, by circumventing human subjectivity and error. While recent breakthroughs in AI technology might facilitate such automation, this has not been demonstrated at scale. It this paper, we introduce a novel automatic short answer grading (ASAG) system. The system is based on a fine-tuned open-source transformer model which we trained on large set of exam data from university courses across a large range of disciplines. We evaluated the trained model's performance against held-out test data in a first experiment and found high accuracy levels across a broad spectrum of unseen questions, even in unseen courses. We further compared the performance of our model with that of certified human domain experts in a second experiment: we first assembled another test dataset from real historical exams - the historic grades contained in that data were awarded to students in a regulated, legally binding examination process; we therefore considered them as ground truth for our experiment. We then asked certified human domain experts and our model to grade the historic student answers again without disclosing the historic grades. Finally, we compared the hence obtained grades with the historic grades (our ground truth). We found that for the courses examined, the model deviated less from the official historic grades than the human re-graders - the model's median absolute error was 44 % smaller than the human re-graders', implying that the model is more consistent than humans in grading. These results suggest that leveraging AI enhanced grading can reduce human subjectivity, improve consistency and thus ultimately increase fairness.

Beyond human subjectivity and error: a novel AI grading system

TL;DR

The paper presents ASAG, a novel automatic short-answer grading system built on a fine-tuned open-source transformer trained on a large, cross-disciplinary university exam dataset. It demonstrates strong generalization to unseen questions and courses, achieving MAE around 1.32–1.44 points, RMSE around 2.27–2.41, and Pearson correlations near 0.69–0.78 on held-out sets, and outperforms human re-graders in matching official grades in a large-scale human benchmark. The study also outlines a concrete architecture, a four-element input/output formulation, and a rigorous evaluation framework including normalization, diverse splits, and a detailed human-vs-AI comparison, underscoring the potential of AI-assisted or autonomous grading. Practical implications include safer deployment via assisted grading levels, explainability enhancements, and a roadmap toward fairer, more consistent grading while reducing educator workload.

Abstract

The grading of open-ended questions is a high-effort, high-impact task in education. Automating this task promises a significant reduction in workload for education professionals, as well as more consistent grading outcomes for students, by circumventing human subjectivity and error. While recent breakthroughs in AI technology might facilitate such automation, this has not been demonstrated at scale. It this paper, we introduce a novel automatic short answer grading (ASAG) system. The system is based on a fine-tuned open-source transformer model which we trained on large set of exam data from university courses across a large range of disciplines. We evaluated the trained model's performance against held-out test data in a first experiment and found high accuracy levels across a broad spectrum of unseen questions, even in unseen courses. We further compared the performance of our model with that of certified human domain experts in a second experiment: we first assembled another test dataset from real historical exams - the historic grades contained in that data were awarded to students in a regulated, legally binding examination process; we therefore considered them as ground truth for our experiment. We then asked certified human domain experts and our model to grade the historic student answers again without disclosing the historic grades. Finally, we compared the hence obtained grades with the historic grades (our ground truth). We found that for the courses examined, the model deviated less from the official historic grades than the human re-graders - the model's median absolute error was 44 % smaller than the human re-graders', implying that the model is more consistent than humans in grading. These results suggest that leveraging AI enhanced grading can reduce human subjectivity, improve consistency and thus ultimately increase fairness.
Paper Structure (31 sections, 6 figures, 10 tables)

This paper contains 31 sections, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Results
  4. Model setup and training
  5. System Architecture
  6. Fine-tuning Approach
  7. Experiment 1: Evaluation on held-out test sets
  8. Raw Grade Performance
  9. Normalized Grade Performance
  10. Performance Split by Maximum Number of Points
  11. Experiment 2: Comparing human against model performance in a grading task
  12. Comparing the Model to Human Re-Graders
  13. Dealing with strongly deviating re-graders
  14. Discussion
  15. Future Directions
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Model inputs and outputs. The model receives a question, a reference answer, the student answer and the maximum number of points as inputs and provides a grade (the number of points for the student answer) as an output.
  • Figure 2: Visualisation of the relevant scores. We compared the deviation of model grades (left) and human regrader grades (right) from official exam grades (middle).
  • Figure 3: Results of the human benchmark experiment. We show all three grades (green: official grade – considered to be the ground truth, blue: human regrader grade, orange: model grade) on the y-axis, mapped against the official grade on the x-axis. The model grades are closer to the official grades (the ground truth) than the human re-grader grades.
  • Figure 4: Re-grader deviations per course. We show RMSE between human re-graders and official grades per course. The ‘extreme’ re-graders that we excluded from the analysis in this section are coloured in dark grey.
  • Figure A1: Error distributions from the human benchmark experiment. We show the distribution of absolute grading errors for both the human regraders (left, blue) and the model (right, orange). The median absolute errors are 20.0 and 11.1 respectively. The model’s median absolute error is hence 44% smaller than the humans’ median absolute error. The distributions differ significantly (p > 0.001, Mann-Whitney U test).
  • ...and 1 more figures