Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoring

TL;DR

This paper tackles the ethical blind spots of Automated Essay Scoring (AES) and Large Language Models (LLMs) by introducing the Harmful Essay Detection (HED) benchmark, which combines argumentative and harmful essays on sensitive topics. It systematically evaluates LLMs and AES models on two tasks: classifying essays as argumentative or harmful and scoring them holistically, while examining the effects of prompting, personas, and name-based biases. The findings show that many models struggle to distinguish harmful content from valid argumentation, and harmful essays can be scored higher than benign ones when ethical considerations are not integrated into scoring rules. The authors propose incorporating harmful-essay annotation guidelines into scoring instructions to improve alignment with human judgment and highlight the need for ethically aware AES systems. Overall, HED provides a rigorous framework and empirical evidence calling for stronger ethical safeguards in automated scoring and content generation in education.

Abstract

This study addresses critical gaps in Automated Essay Scoring (AES) systems and Large Language Models (LLMs) with regard to their ability to effectively identify and score harmful essays. Despite advancements in AES technology, current models often overlook ethically and morally problematic elements within essays, erroneously assigning high scores to essays that may propagate harmful opinions. In this study, we introduce the Harmful Essay Detection (HED) benchmark, which includes essays integrating sensitive topics such as racism and gender bias, to test the efficacy of various LLMs in recognizing and scoring harmful content. Our findings reveal that: (1) LLMs require further enhancement to accurately distinguish between harmful and argumentative essays, and (2) both current AES models and LLMs fail to consider the ethical dimensions of content during scoring. The study underscores the need for developing more robust AES systems that are sensitive to the ethical implications of the content they are scoring.

Figures (7)

• Figure 1: Examples of harmful essays from our HED benchmark and the results of their classification and scoring by an existing AES model (BERT-based, trained on the IELTS dataset with a scoring scale of 1 to 9) and various LLMs.
• Figure 2: Toxicity comparison between essays in the IELTS dataset and those in our HED benchmark. The toxicity scale ranges from 0 to 1, with higher values indicating greater toxicity.
• Figure 3: Correlation between POR and performance of essay classification.
• Figure 4: Results of various racial persona instructions for different LLMs. Scores for each model were averaged over three trials. None indicates the results using the basic classifying instruction.
• Figure 5: Results of various personality-based instructions for different LLMs. Scores for each model were averaged over three trials.