Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP

TL;DR

This paper investigates machine-assisted grading of nationwide school-leaving essays in Estonian using zero-shot LLMs and supervised NLP features. It analyzes a large, real-world dataset from Estonian trial exams, comparing LLM outputs and feature-based models against human consensus scores under a rubric-guided, human-in-the-loop framework. The findings show that LLMs can achieve near-human consistency and that supervised features provide complementary gains in certain rubric areas, while highlighting risks such as prompt injection and the potential for LLMs to generate high-scoring but non-authentic essays. The study advocates a modular, auditable deployment strategy that preserves human accountability, enables scalable feedback, and aligns with evolving regulatory standards, illustrating a viable path for national-scale, small-language assessment systems.

Abstract

Large language models (LLMs) enable rapid and consistent automated evaluation of open-ended exam responses, including dimensions of content and argumentation that have traditionally required human judgment. This is particularly important in cases where a large amount of exams need to be graded in a limited time frame, such as nation-wide graduation exams in various countries. Here, we examine the applicability of automated scoring on two large datasets of trial exam essays of two full national cohorts from Estonia. We operationalize the official curriculum-based rubric and compare LLM and statistical natural language processing (NLP) based assessments with human panel scores. The results show that automated scoring can achieve performance comparable to that of human raters and tends to fall within the human scoring range. We also evaluate bias, prompt injection risks, and LLMs as essay writers. These findings demonstrate that a principled, rubric-driven, human-in-the-loop scoring pipeline is viable for high-stakes writing assessment, particularly relevant for digitally advanced societies like Estonia, which is about to adapt a fully electronic examination system. Furthermore, the system produces fine-grained subscore profiles that can be used to generate systematic, personalized feedback for instruction and exam preparation. The study provides evidence that LLM-assisted assessment can be implemented at a national scale, even in a small-language context, while maintaining human oversight and compliance with emerging educational and regulatory standards.

Figures (5)

• Figure 1: Overview of the two datasets: average score, standard deviations between grader scores, and essay lengths, as "beeswarm" graphs where points are pushed aside to avoid overlap, illustrating the distributions. Each essay is one dot, colored by student sex (as this was available in the metadata). Black bars are averages. With the exception of some outliers, the samples are fairly realistic in terms of scores, what is known about grading challenges in terms of inter-grader agreement, and length, with most students following the guidelines of their respective tasks (therefore, longer texts in 12th grade).
• Figure 2: Results of the grading experiments, rubric category by category (horizontal axes), for both the 9th (panel A) and 12th grade (B). The model mean absolute error is on the vertical axis, and has the same 0-3 scale as the grading rubric, with colored bars indicating different models. The gray horizontal lines indicate how much the two human graders deviate from their own consensus (average). Given there are only two graders, they also function as a measure of human variance. The black notches provide another error measurement, distance outside the range of the human scores. Error bars and notches close to or below the gray lines indicate essentially near-human performance. The statistical NLP pipeline results are means from the cross-validation, with small standard deviations in the 0.03-0.06 range. The ample white space in the figure indicates a promising result: the errors are relatively small across the categories and most models, and often within the range of human grading variation.
• Figure 3: Injection experiment results. A simple grading prompt without any protection against prompt injection is vulnerable to having its original guidance overridden by strongly imperative language in the part of the input meant for analysis or assessment. Each dot is an essay, colored by the difference between the injected and regular prompt, persistently across all three groups for comparison. With the exception of a few essays (red dots), the gains are small, but never zero.
• Figure 4: One possible high-level framework for the integration of machines-assisted grading into high-stake large-scale examination.
• Figure 5: This graph complements the results graph in the main text, offering an alternative view into the results as agreement accuracy, instead of mean absolute error. Accuracy here is simply recalculated as $100 * (1 - \mathrm{MAE}/3))$, so higher values indicate better performance (higher agreement between machines and humans).