A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

TL;DR

This work tackles the challenge of analyzing rich, multimodal data from children's embodied learning in mixed-reality by embedding machine learning into Interaction Analysis (IA). It introduces an interactive visual timeline that fuses movement, gaze, affect, and system-state data with IA insights to support researchers in identifying key learning moments during a photosynthesis task. The GEM-STEP study demonstrates the feasibility of 3D gaze encoding, affective state labeling, and 3D room reconstruction to map attention and engagement within a MR environment. The proposed AI-in-the-loop timeline enables more efficient, scalable IA, supports mixed-methods analysis, and offers a path toward extending to additional models and modalities for embodied learning research.

Abstract

Investigating children's embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students' learning patterns. Our study aims to simplify researchers' tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students' scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students' states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students' temporal learning progressions.

Figures (3)

• Figure 1: Overview of Affect Detection Process
• Figure 2: Gaze Estimation Pipeline.
• Figure 3: Visual timeline of multimodal data for IA