SensEmo: Enabling Affective Learning through Real-time Emotion Recognition with Smartwatches

TL;DR

SensEmo addresses the need to monitor and adapt to student emotions in learning environments by leveraging physiological signals from commercial smartwatches. It combines a personalized valence-arousal emotion model with a reinforcement-learning–driven MDP controller to adjust teaching content and pacing in real time. The system achieves 88.9% accuracy in emotion recognition and demonstrates substantial learning gains, including a 40% increase in online quiz scores, across online and in-person settings. By validating a wearable-based affective learning loop in realistic classrooms, the work highlights practical feasibility and potential for broad impact in education.

Abstract

Recent research has demonstrated the capability of physiological signals to infer both user emotional and attention responses. This presents an opportunity for leveraging widely available physiological sensors in smartwatches, to detect real-time emotional cues in users, such as stress and excitement. In this paper, we introduce SensEmo, a smartwatch-based system designed for affective learning. SensEmo utilizes multiple physiological sensor data, including heart rate and galvanic skin response, to recognize a student's motivation and concentration levels during class. This recognition is facilitated by a personalized emotion recognition model that predicts emotional states based on degrees of valence and arousal. With real-time emotion and attention feedback from students, we design a Markov decision process-based algorithm to enhance student learning effectiveness and experience by by offering suggestions to the teacher regarding teaching content and pacing. We evaluate SensEmo with 22 participants in real-world classroom environments. Evaluation results show that SensEmo recognizes student emotion with an average of 88.9% accuracy. More importantly, SensEmo assists students to achieve better online learning outcomes, e.g., an average of 40.0% higher grades in quizzes, over the traditional learning without student emotional feedback.

Figures (7)

• Figure 1: SensEmo uses physiological signals to personalize the emotion model and adapt the learning content and pace to the student's emotional state.
• Figure 2: Overview of the affective learning feedback controller, including emotion recognition and a reinforcement learning-based controller.
• Figure 3: The 2-dimensional valence-arousal space and mapping of emotions of bored, satisfied, curious, and confused.
• Figure 4: The discrete MDP representation of SensEmo.
• Figure 5: Convergent asymptotic behavior of the MDP.