Artificial Intelligence Bias on English Language Learners in Automatic Scoring

TL;DR

The paper investigates whether AI-based automatic scoring introduces bias against English Language Learners (ELLs) in science constructed-response items. By fine-tuning BERT on four training regimes (ELL-only, Non-ELL-only, Unbalanced Mixed, Balanced Mixed) and evaluating 21 CAST Grade 8 items across three data-scale groups, the study assesses bias via accuracy and disparities via Mean Score Gap (MSG), using Friedman and Wilcoxon tests. Key finding: with sufficiently large training data ($n_{ELL}\approx30{,}000$ or $n_{ELL}\approx1{,}000$), AI scoring shows no significant bias or distorted disparities between ELL and non-ELL students; with very small samples ($n_{ELL}\approx200$), biases may emerge, though representative proportions in training data mitigate this. The results suggest that AI scoring for science can be unbiased when data are ample and representative, but caution is warranted for limited data, and future work should address broader bias types and proficiency-level granularity to support inclusive assessments.

Abstract

This study investigated potential scoring biases and disparities toward English Language Learners (ELLs) when using automatic scoring systems for middle school students' written responses to science assessments. We specifically focus on examining how unbalanced training data with ELLs contributes to scoring bias and disparities. We fine-tuned BERT with four datasets: responses from (1) ELLs, (2) non-ELLs, (3) a mixed dataset reflecting the real-world proportion of ELLs and non-ELLs (unbalanced), and (4) a balanced mixed dataset with equal representation of both groups. The study analyzed 21 assessment items: 10 items with about 30,000 ELL responses, five items with about 1,000 ELL responses, and six items with about 200 ELL responses. Scoring accuracy (Acc) was calculated and compared to identify bias using Friedman tests. We measured the Mean Score Gaps (MSGs) between ELLs and non-ELLs and then calculated the differences in MSGs generated through both the human and AI models to identify the scoring disparities. We found that no AI bias and distorted disparities between ELLs and non-ELLs were found when the training dataset was large enough (ELL = 30,000 and ELL = 1,000), but concerns could exist if the sample size is limited (ELL = 200).

Figures (1)

• Figure 1: Analyzing disparity by comparing MSGs between human and machine scores.