Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Assessment of Students' Classroom Engagement with Bias Mitigated Multi-task Model

James Thiering, Tarun Sethupat Radha Krishna, Dylan Zelkin, Ashis Kumer Biswas

TL;DR

The paper tackles bias in automated engagement assessment for online classrooms by introducing a bias-mitigated training framework that uses attribute-orthogonal regularization in a split-model, multi-task setting. It leverages an external gender classifier pretrained on a separate dataset to minimize gender leakage into engagement predictions, achieving a substantial reduction in gender-based prediction disparity as evidenced by the Pearson correlation coefficient improving from $0.897$ to $0.999$. The approach is evaluated on the DAiSEE dataset, with analyses including Grad-CAM visualization and Fairlearn F1 metrics, highlighting both improvements in fairness and class-level trade-offs due to ground-truth skew. The work demonstrates that debiasing can be achieved through architectural design and regularization rather than solely data manipulation, with implications for fairer automated feedback in online education and potential applicability to other vision-based sensitive-prediction tasks.

Abstract

With the rise of online and virtual learning, monitoring and enhancing student engagement have become an important aspect of effective education. Traditional methods of assessing a student's involvement might not be applicable directly to virtual environments. In this study, we focused on this problem and addressed the need to develop an automated system to detect student engagement levels during online learning. We proposed a novel training method which can discourage a model from leveraging sensitive features like gender for its predictions. The proposed method offers benefits not only in the enforcement of ethical standards, but also to enhance interpretability of the model predictions. We applied an attribute-orthogonal regularization technique to a split-model classifier, which uses multiple transfer learning strategies to achieve effective results in reducing disparity in the distribution of prediction for sensitivity groups from a Pearson correlation coefficient of 0.897 for the unmitigated model, to 0.999 for the mitigated model. The source code for this project is available on https://github.com/ashiskb/elearning-engagement-study .

Automatic Assessment of Students' Classroom Engagement with Bias Mitigated Multi-task Model

TL;DR

The paper tackles bias in automated engagement assessment for online classrooms by introducing a bias-mitigated training framework that uses attribute-orthogonal regularization in a split-model, multi-task setting. It leverages an external gender classifier pretrained on a separate dataset to minimize gender leakage into engagement predictions, achieving a substantial reduction in gender-based prediction disparity as evidenced by the Pearson correlation coefficient improving from to . The approach is evaluated on the DAiSEE dataset, with analyses including Grad-CAM visualization and Fairlearn F1 metrics, highlighting both improvements in fairness and class-level trade-offs due to ground-truth skew. The work demonstrates that debiasing can be achieved through architectural design and regularization rather than solely data manipulation, with implications for fairer automated feedback in online education and potential applicability to other vision-based sensitive-prediction tasks.

Abstract

With the rise of online and virtual learning, monitoring and enhancing student engagement have become an important aspect of effective education. Traditional methods of assessing a student's involvement might not be applicable directly to virtual environments. In this study, we focused on this problem and addressed the need to develop an automated system to detect student engagement levels during online learning. We proposed a novel training method which can discourage a model from leveraging sensitive features like gender for its predictions. The proposed method offers benefits not only in the enforcement of ethical standards, but also to enhance interpretability of the model predictions. We applied an attribute-orthogonal regularization technique to a split-model classifier, which uses multiple transfer learning strategies to achieve effective results in reducing disparity in the distribution of prediction for sensitivity groups from a Pearson correlation coefficient of 0.897 for the unmitigated model, to 0.999 for the mitigated model. The source code for this project is available on https://github.com/ashiskb/elearning-engagement-study .
Paper Structure (22 sections, 2 equations, 12 figures, 1 table)

This paper contains 22 sections, 2 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Transfer Learning in Deep CNN Models
  4. Generalized Additive Models
  5. Sample Reweight
  6. Adversarial Models
  7. Split-Model Design
  8. The dataset and associated challenges
  9. Temporal Consideration
  10. Label Skew
  11. Bias and Uncertainty in Ground Truth
  12. Human Validation
  13. Proposed Methodology
  14. Model Architecture
  15. Grad-CAM
  16. ...and 7 more sections

Figures (12)

  • Figure 1: Adversarial-trained feature suppression from wang2019balanced
  • Figure 2: Samples of Different Engagement Levels. Top row: very-low engagement; Second row: low engagement; Third row: high engagement; Bottom row: very-high engagement.
  • Figure 3: Sample frames displaying contradicting engagement levels
  • Figure 4: Distribution of labels
  • Figure 5: Distribution of labels, separated by gender
  • ...and 7 more figures