Multimodal Engagement Analysis from Facial Videos in the Classroom

TL;DR

This work collected audiovisual recordings of secondary school classes over a one and a half month period, acquired continuous engagement labeling per student in repeated sessions, and explored computer vision methods to classify engagement from facial videos.

Abstract

Student engagement is a key construct for learning and teaching. While most of the literature explored the student engagement analysis on computer-based settings, this paper extends that focus to classroom instruction. To best examine student visual engagement in the classroom, we conducted a study utilizing the audiovisual recordings of classes at a secondary school over one and a half month's time, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement levels from faces in the classroom. We trained deep embeddings for attentional and emotional features, training Attention-Net for head pose estimation and Affect-Net for facial expression recognition. We additionally trained different engagement classifiers, consisting of Support Vector Machines, Random Forest, Multilayer Perceptron, and Long Short-Term Memory, for both features. The best performing engagement classifiers achieved AUCs of .620 and .720 in Grades 8 and 12, respectively. We further investigated fusion strategies and found score-level fusion either improves the engagement classifiers or is on par with the best performing modality. We also investigated the effect of personalization and found that using only 60-seconds of person-specific data selected by margin uncertainty of the base classifier yielded an average AUC improvement of .084. 4.Our main aim with this work is to provide the technical means to facilitate the manual data analysis of classroom videos in research on teaching quality and in the context of teacher training.

Figures (7)

• Figure 1: Sample scene from the classroom. The synchronous cameras recorded the instruction simultaneously.
• Figure 2: Continuous scale of our manual rating instrument and visible behavioral indicators Goldberg:2019
• Figure 3: The distribution of engagement labels in Grade 8 and 12. Pie charts show the percentage of quantized labels according to continuous labelling.
• Figure 4: Feature learning for affect and attention. Two ResNet-50 backbones are separately trained for facial expression recognition and head pose estimation. The learned features subsequently will be used for engagement estimation on classroom data.
• Figure 5: Batch-mode Active Learning for Personalized Engagement Classification (The initial network is the engagement classifier trained in a person-independent manner and the weights of the feature extractors kept frozen during all experiments).