Context Matters: Peer-Aware Student Behavioral Engagement Measurement via VLM Action Parsing and LLM Sequence Classification

TL;DR

This paper tackles the challenge of measuring student behavioral engagement in classrooms while accounting for the crucial role of peer context. It introduces a three-stage framework that first performs few-shot action recognition with a vision-language model, then parses continuous actions into a time-ordered sequence via sliding-window segmentation, and finally uses a large language model to classify engagement by incorporating classroom context. A novel dual-component dataset is created to support both action labeling and engagement annotation, and the approach demonstrates that temporal sequences and peer context outperform simple histograms or context-free baselines in identifying engaged versus disengaged students. The results show strong engagement classification performance (up to high F1 scores) and reveal that context awareness substantially improves interpretation of student actions, offering a scalable path to privacy-preserving, data-efficient engagement assessment in real classrooms.

Abstract

Understanding student behavior in the classroom is essential to improve both pedagogical quality and student engagement. Existing methods for predicting student engagement typically require substantial annotated data to model the diversity of student behaviors, yet privacy concerns often restrict researchers to their own proprietary datasets. Moreover, the classroom context, represented in peers' actions, is ignored. To address the aforementioned limitation, we propose a novel three-stage framework for video-based student engagement measurement. First, we explore the few-shot adaptation of the vision-language model for student action recognition, which is fine-tuned to distinguish among action categories with a few training samples. Second, to handle continuous and unpredictable student actions, we utilize the sliding temporal window technique to divide each student's 2-minute-long video into non-overlapping segments. Each segment is assigned an action category via the fine-tuned VLM model, generating a sequence of action predictions. Finally, we leverage the large language model to classify this entire sequence of actions, together with the classroom context, as belonging to an engaged or disengaged student. The experimental results demonstrate the effectiveness of the proposed approach in identifying student engagement.

Figures (7)

• Figure 1: Context matters in engagement classification. Under traditional approaches, a student’s action sequence showing continuous typing on a laptop may be labeled as engaged. However, our peer-aware model classifies the same sequence as disengaged when surrounding students are not engaged in similar note-taking behavior.
• Figure 2: Overview of the proposed framework. Students' tubelets are extracted from a two-minute video, and each student's tube is divided into non-overlapping segments using a sliding temporal window technique. The action sequence is then produced by the fine-tuned VLM model. LLM uses the generated action sequence to classify engagement after integrating it with peers' action sequences.
• Figure 3: a) for engaged student, b, c) for disengaged student
• Figure 4: Samples of students' actions during class time.
• Figure 6: An example of the task description $x_{desc}$