Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?

TL;DR

The paper examines whether prompt-based LLMs used for Automated Essay Scoring can infer student demographics from essays and how such inferences relate to scoring bias. Using GPT-4o on the PERSUADE 2.0 corpus, the authors design prompts to predict gender and first-language background and to score essays, then analyze the relationship between demographic predictability and scoring fairness via inverse-probability-weighted regression. They find that LLMs can infer demographics with high accuracy for L1 and substantial accuracy for gender when predictions are definitive, and that scoring bias is more pronounced when the model correctly predicts L1—especially harming non-native speakers—while gender bias remains largely insignificant. The work suggests that demographic cues are embedded in prompt-based AES and that debiasing in prompt-based settings may require strategies beyond standard fine-tuning, such as careful prompt design and sample diversity. Overall, the study highlights fairness concerns in prompt-based AES and provides a framework to quantify and address demographic-related scoring biases in educational NLP applications.

Abstract

Large Language Models (LLMs) are widely used in Automated Essay Scoring (AES) due to their ability to capture semantic meaning. Traditional fine-tuning approaches required technical expertise, limiting accessibility for educators with limited technical backgrounds. However, prompt-based tools like ChatGPT have made AES more accessible, enabling educators to obtain machine-generated scores using natural-language prompts (i.e., the prompt-based paradigm). Despite advancements, prior studies have shown bias in fine-tuned LLMs, particularly against disadvantaged groups. It remains unclear whether such biases persist or are amplified in the prompt-based paradigm with cutting-edge tools. Since such biases are believed to stem from the demographic information embedded in pre-trained models (i.e., the ability of LLMs' text embeddings to predict demographic attributes), this study explores the relationship between the model's predictive power of students' demographic attributes based on their written works and its predictive bias in the scoring task in the prompt-based paradigm. Using a publicly available dataset of over 25,000 students' argumentative essays, we designed prompts to elicit demographic inferences (i.e., gender, first-language background) from GPT-4o and assessed fairness in automated scoring. Then we conducted multivariate regression analysis to explore the impact of the model's ability to predict demographics on its scoring outcomes. Our findings revealed that (i) prompt-based LLMs can somewhat infer students' demographics, particularly their first-language backgrounds, from their essays; (ii) scoring biases are more pronounced when the LLM correctly predicts students' first-language background than when it does not; and (iii) scoring error for non-native English speakers increases when the LLM correctly identifies them as non-native.