Grade Like a Human: Rethinking Automated Assessment with Large Language Models

TL;DR

This paper tackles the problem of automated short-answer grading by highlighting the limitations of existing LLM-based systems that rely on fixed rubrics. It proposes Grade-Like-a-Human, a three-stage, multi-agent framework comprising rubric generation, grading, and post-grading review, with rubric refinement guided by student answers and a group-based review mechanism to ensure fairness and consistency. The authors introduce the OS dataset for evaluating holistic grading and demonstrate improvements in grading accuracy and reliability on OS and Mohler datasets, validating the benefits of adaptive rubrics and cross-stage collaboration. The work offers practical design principles for deploying LLM-based assessment systems and provides open resources to spur further research in automated, fair, and robust grading.

Abstract

While large language models (LLMs) have been used for automated grading, they have not yet achieved the same level of performance as humans, especially when it comes to grading complex questions. Existing research on this topic focuses on a particular step in the grading procedure: grading using predefined rubrics. However, grading is a multifaceted procedure that encompasses other crucial steps, such as grading rubrics design and post-grading review. There has been a lack of systematic research exploring the potential of LLMs to enhance the entire grading~process. In this paper, we propose an LLM-based grading system that addresses the entire grading procedure, including the following key components: 1) Developing grading rubrics that not only consider the questions but also the student answers, which can more accurately reflect students' performance. 2) Under the guidance of grading rubrics, providing accurate and consistent scores for each student, along with customized feedback. 3) Conducting post-grading review to better ensure accuracy and fairness. Additionally, we collected a new dataset named OS from a university operating system course and conducted extensive experiments on both our new dataset and the widely used Mohler dataset. Experiments demonstrate the effectiveness of our proposed approach, providing some new insights for developing automated grading systems based on LLMs.

Figures (9)

• Figure 1: Overview of the Grade-Like-a-Human system
• Figure 2: One-shot prompt
• Figure 3: Self-reflection prompt
• Figure 4: Batching prompt
• Figure 5: Our proposed re-grouping strategy.