Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Bias in Automated Grading Systems for ESL Learners: A Contrastive Learning Approach

Kevin Fan, Eric Yun

TL;DR

This work targets bias against ESL learners in transformer-based Automated Essay Scoring by identifying a high-proficiency ESL penalty in a DeBERTa-v3 baseline. It introduces Contrastive Learning with Matched Essay Pairs using Triplet Margin Loss to align ESL and Native embeddings, with a dataset of 17,161 triplets formed from ASAP 2.0 and ELLIPSE. The approach reduces the high-proficiency ESL gap from $0.103$ to $0.062$ while maintaining a QWK of $0.756$, and post-hoc analysis shows the model disentangles sentence complexity from grammatical errors. This demonstrates that a fairness-focused contrastive objective can improve equity in AES without prohibitive losses in accuracy, guiding deployment considerations and future fairness research across demographics and architectures.

Abstract

As Automated Essay Scoring (AES) systems are increasingly used in high-stakes educational settings, concerns regarding algorithmic bias against English as a Second Language (ESL) learners have increased. Current Transformer-based regression models trained primarily on native-speaker corpora often learn spurious correlations between surface-level L2 linguistic features and essay quality. In this study, we conduct a bias study of a fine-tuned DeBERTa-v3 model using the ASAP 2.0 and ELLIPSE datasets, revealing a constrained score scaling for high-proficiency ESL writing where high-proficiency ESL essays receive scores 10.3% lower than Native speaker essays of identical human-rated quality. To mitigate this, we propose applying contrastive learning with a triplet construction strategy: Contrastive Learning with Matched Essay Pairs. We constructed a dataset of 17,161 matched essay pairs and fine-tuned the model using Triplet Margin Loss to align the latent representations of ESL and Native writing. Our approach reduced the high-proficiency scoring disparity by 39.9% (to a 6.2% gap) while maintaining a Quadratic Weighted Kappa (QWK) of 0.76. Post-hoc linguistic analysis suggests the model successfully disentangled sentence complexity from grammatical error, preventing the penalization of valid L2 syntactic structures.

Mitigating Bias in Automated Grading Systems for ESL Learners: A Contrastive Learning Approach

TL;DR

This work targets bias against ESL learners in transformer-based Automated Essay Scoring by identifying a high-proficiency ESL penalty in a DeBERTa-v3 baseline. It introduces Contrastive Learning with Matched Essay Pairs using Triplet Margin Loss to align ESL and Native embeddings, with a dataset of 17,161 triplets formed from ASAP 2.0 and ELLIPSE. The approach reduces the high-proficiency ESL gap from to while maintaining a QWK of , and post-hoc analysis shows the model disentangles sentence complexity from grammatical errors. This demonstrates that a fairness-focused contrastive objective can improve equity in AES without prohibitive losses in accuracy, guiding deployment considerations and future fairness research across demographics and architectures.

Abstract

As Automated Essay Scoring (AES) systems are increasingly used in high-stakes educational settings, concerns regarding algorithmic bias against English as a Second Language (ESL) learners have increased. Current Transformer-based regression models trained primarily on native-speaker corpora often learn spurious correlations between surface-level L2 linguistic features and essay quality. In this study, we conduct a bias study of a fine-tuned DeBERTa-v3 model using the ASAP 2.0 and ELLIPSE datasets, revealing a constrained score scaling for high-proficiency ESL writing where high-proficiency ESL essays receive scores 10.3% lower than Native speaker essays of identical human-rated quality. To mitigate this, we propose applying contrastive learning with a triplet construction strategy: Contrastive Learning with Matched Essay Pairs. We constructed a dataset of 17,161 matched essay pairs and fine-tuned the model using Triplet Margin Loss to align the latent representations of ESL and Native writing. Our approach reduced the high-proficiency scoring disparity by 39.9% (to a 6.2% gap) while maintaining a Quadratic Weighted Kappa (QWK) of 0.76. Post-hoc linguistic analysis suggests the model successfully disentangled sentence complexity from grammatical error, preventing the penalization of valid L2 syntactic structures.
Paper Structure (16 sections, 1 equation, 3 figures, 1 table)

This paper contains 16 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Preparation
  4. Baseline Model
  5. Fairness Triplet Construction
  6. Contrastive Training
  7. Validation and Scoring
  8. Results
  9. Phase 1: Baseline Testing
  10. Phase 2: Mitigation Performance
  11. Ablation Study
  12. Discussion
  13. Linguistic Triggers
  14. Limitations
  15. Scope and Future
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Vector-Based Scoring Architecture. The model $f_\theta$ maps the input tokens to a latent vector $h$. The regression head projects $h$ to a score. Bias occurs when $f_\theta$ learns to encode surface-level heuristics (shortcuts) into $h$ rather than true semantic quality.
  • Figure 2: Triplet Loss Visualization. The loss function minimizes the distance between an anchor and a positive (same identity/score) while maximizing the distance between the anchor and a negative (different identity/score).
  • Figure 3: Fairness vs. Accuracy Trade-off. The proposed model (Red Star) moves significantly to the right (fairer) with only a minor drop in accuracy compared to the Baseline (Blue Square).