Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education

TL;DR

The paper asks whether LLM-based evaluators can assess human-written student texts for education, extending prior machine-generated evaluation methods. It collects 100 texts from 32 Korean students across 15 writing types and uses GPT-4-Turbo to judge five criteria (grammaticality, fluency, coherence, consistency, relevance), with students subsequently validating the judgments. Results show LLMs are reasonably reliable for grammaticality and fluency and for more objective writing, but less reliable for subjective criteria and informal styles, with age-related differences in scoring. The study provides valuable educational feedback implications and releases the dataset for future research, while highlighting gaps in evaluating subjective human writing. Overall, the work lays groundwork for integrating LLM-based feedback into student writing improvement, with caveats and directions for more robust evaluation frameworks.

Abstract

Large language model (LLM)-based evaluation pipelines have demonstrated their capability to robustly evaluate machine-generated text. Extending this methodology to assess human-written text could significantly benefit educational settings by providing direct feedback to enhance writing skills, although this application is not straightforward. In this paper, we investigate whether LLMs can effectively assess human-written text for educational purposes. We collected 100 texts from 32 Korean students across 15 types of writing and employed GPT-4-Turbo to evaluate them using grammaticality, fluency, coherence, consistency, and relevance as criteria. Our analyses indicate that LLM evaluators can reliably assess grammaticality and fluency, as well as more objective types of writing, though they struggle with other criteria and types of writing. We publicly release our dataset and feedback.