ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues

TL;DR

ViBED-Net tackles the challenge of detecting student engagement from video by combining face-focused and scene-focused spatial features via two EfficientNetV2 streams and modeling temporal dynamics with both LSTM and Transformer encoders. The LSTM-based variant achieves a new state-of-the-art accuracy of 73.43% on the DAiSEE dataset, outperforming Transformer and prior approaches by effectively capturing nuanced temporal patterns in engagement. Key contributions include the dual-stream architecture, explicit handling of class imbalance through data augmentation, and a modular design adaptable to education, UX research, and personalization. The work advances video-based affective computing by delivering a scalable framework that leverages both facial cues and contextual scene information for robust engagement analysis, with practical implications for online learning analytics and adaptive content delivery.

Abstract

Engagement detection in online learning environments is vital for improving student outcomes and personalizing instruction. We present ViBED-Net (Video-Based Engagement Detection Network), a novel deep learning framework designed to assess student engagement from video data using a dual-stream architecture. ViBED-Net captures both facial expressions and full-scene context by processing facial crops and entire video frames through EfficientNetV2 for spatial feature extraction. These features are then analyzed over time using two temporal modeling strategies: Long Short-Term Memory (LSTM) networks and Transformer encoders. Our model is evaluated on the DAiSEE dataset, a large-scale benchmark for affective state recognition in e-learning. To enhance performance on underrepresented engagement classes, we apply targeted data augmentation techniques. Among the tested variants, ViBED-Net with LSTM achieves 73.43\% accuracy, outperforming existing state-of-the-art approaches. ViBED-Net demonstrates that combining face-aware and scene-aware spatiotemporal cues significantly improves engagement detection accuracy. Its modular design allows flexibility for application across education, user experience research, and content personalization. This work advances video-based affective computing by offering a scalable, high-performing solution for real-world engagement analysis. The source code for this project is available on https://github.com/prateek-gothwal/ViBED-Net .

Figures (4)

• Figure 1: Frames from DAiSEE dataset
• Figure 2: ViBED-Net Architecture
• Figure 3: MLP Architecture
• Figure 4: Confusion matrix EfficientNet+LSTM on DAiSEE test set.