AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy

TL;DR

The paper tackles scalable automated grading of short-answer constructed-response questions by leveraging near-domain data. It introduces a framework that fine-tunes BERT-based models on related questions within the same topic, enabling competitive accuracy with substantially less labeled data and outperforming non-fine-tuned large language models like GPT-3.5 and GPT-4 in key settings. A detailed analysis reveals data- and accuracy-advantages, model-size considerations, and practical implications for classroom feedback, while acknowledging subjectivity in human labels and domain-specific limitations. The findings suggest near-domain data-driven fine-tuning as a practical path toward fair, efficient, and scalable automated grading across educational contexts, with future work extending to additional domains and larger models.

Abstract

Constructed-response questions are crucial to encourage generative processing and test a learner's understanding of core concepts. However, the limited availability of instructor time, large class sizes, and other resource constraints pose significant challenges in providing timely and detailed evaluation, which is crucial for a holistic educational experience. In addition, providing timely and frequent assessments is challenging since manual grading is labor intensive, and automated grading is complex to generalize to every possible response scenario. This paper proposes a novel and practical approach to grade short-answer constructed-response questions. We discuss why this problem is challenging, define the nature of questions on which our method works, and finally propose a framework that instructors can use to evaluate their students' open-responses, utilizing near-domain data like data from similar questions administered in previous years. The proposed method outperforms the state of the art machine learning models as well as non-fine-tuned large language models like GPT 3.5, GPT 4, and GPT 4o by a considerable margin of over 10-20% in some cases, even after providing the LLMs with reference/model answers. Our framework does not require pre-written grading rubrics and is designed explicitly with practical classroom settings in mind. Our results also reveal exciting insights about learning from near-domain data, including what we term as accuracy and data advantages using human-labeled data, and we believe this is the first work to formalize the problem of automated short answer grading based on the near-domain data.

Figures (3)

• Figure 1: Model terminology and fine-tuning process: (a) $B_0$ represents the base BERT model, fine-tuned separately on training data $T_1$, $T_2$, $T_3$ to create $BMQ_1$, $BMQ_2$, $BMQ_3$ and (b) Near-domain fine-tuning is applied sequentially where $B_0$ is fine-tuned on $T_1$ to produce $BMQ_1$, $BMQ_1$ is fine-tuned on a subset of $T_2$ to produce $BMQ_1Q_2$, which is further fine-tuned on a subset of $T_3$ to obtain $BMQ_1Q_2Q_3$.
• Figure 2: GPT-based student response classification pipeline:The question type (Replication, Transcription, or Translation), sample correct, incomplete, and incorrect answers, and the student response are added for different cases into the prompt which is then fed into pre-trained GPT models (both fine-tuned and non-fine-tuned versions) to generate classification outputs with Human assistance needed to map the model-generated text to classification categories (Correct, Incomplete, or Incorrect). Domain expert-graded answers serve as the ground truth for evaluation.
• Figure 3: Advantages from Sequential Fine-Tuning with Near-Domain Data: Accuracy versus training data percentage for sequentially fine-tuned $BMQ_1...Q_j$ (j = 2, 3) models on transcription and translation questions. In particular, (a) Accuracy of $BMQ_1Q_2$ on the transcription task as training data increases shown in comparison to the accuracy of $BMQ_2$ and (b) Accuracy of $BMQ_1Q_2Q_3$ on the translation task as training data increases shown in comparison to the accuracy of $BMQ_3$. The results demonstrate that sequential fine-tuning on near-domain data enhances model performance on both tasks, achieving higher accuracy with less training data compared to $BMQ_i$ models. The results also show that the accuracy plateaus off as more data is fed into the model.