Mitigating Data Imbalance in Automated Speaking Assessment

TL;DR

The paper tackles class imbalance in Automated Speaking Assessment by introducing Balancing Logit Variation (BLV) loss, which perturbs class logits during training to expand minority-class feature regions without changing the data. Integrated into a Whisper-transcription and BERT-based classification pipeline, BLV demonstrates improved accuracy, fairness (macro metrics), and discriminability on ICNALE, with larger variance injections (e.g., $\sigma=6$) yielding notable gains. Compared to focal loss, BLV offers a more balanced improvement across PCC, RMSE, and accuracy, supported by a visualization that shows clearer logit clustering. Overall, BLV provides a practical, data-efficient approach to reducing bias in imbalanced ASA tasks and enhances robustness across diverse L2 learners.

Abstract

Automated Speaking Assessment (ASA) plays a crucial role in evaluating second-language (L2) learners proficiency. However, ASA models often suffer from class imbalance, leading to biased predictions. To address this, we introduce a novel objective for training ASA models, dubbed the Balancing Logit Variation (BLV) loss, which perturbs model predictions to improve feature representation for minority classes without modifying the dataset. Evaluations on the ICNALE benchmark dataset show that integrating the BLV loss into a celebrated text-based (BERT) model significantly enhances classification accuracy and fairness, making automated speech evaluation more robust for diverse learners.

Figures (4)

• Figure 1: Model Architecture
• Figure 2: Distribution of ICNALE Dataset on CEFR LEVEL
• Figure 3: Confusion Matrix on ICNALE Testing Set
• Figure 4: t-SNE Visualization of the logits on ICNALE Training Set