Your Eyes Controlled the Game: Real-Time Cognitive Training Adaptation based on Eye-Tracking and Physiological Data in Virtual Reality

TL;DR

This study addresses the need for real-time adaptation of cognitive load in VR cognitive training by deploying a model-driven system that fuses eye-tracking and physiological signals. It introduces an attention-enhanced bidirectional LSTM trained offline and then evaluated in a real-time VR setting, comparing model-guided adaptation to participant-guided control across single- and dual-task paradigms. Key findings show the model can push users to higher difficulty in dual-task conditions without sacrificing weighted performance or increasing subjective workload, and that subjective judgments diverge from objective load, highlighting the value of physiological sensing for objective adaptation. The work demonstrates that automated, zero-calibration cognitive-load management can match traditional, user-driven training in efficacy while offering robust training progression and reduced reliance on subjective self-assessment, with broad implications for home-based VR rehabilitation and education.

Abstract

Cognitive training for sustained attention and working memory is vital across domains relying on robust mental capacity such as education or rehabilitation. Adaptive systems are essential, dynamically matching difficulty to user ability to maintain engagement and accelerate learning. Current adaptive systems often rely on simple performance heuristics or predict visual complexity and affect instead of cognitive load. This study presents the first implementation of real-time adaptive cognitive load control in Virtual Reality cognitive training based on eye-tracking and physiological data. We developed a bidirectional LSTM model with a self-attention mechanism, trained on eye-tracking and physiological (PPG, GSR) data from 74 participants. We deployed it in real-time with 54 participants across single-task (sustained attention) and dual-task (sustained attention + mental arithmetic) paradigms. Difficulty was adjusted dynamically based on participant self-assessment or model's real-time cognitive load predictions. Participants showed a tendency to estimate the task as too difficult, even though they were objectively performing at their best. Over the course of a 10-minute session, both adaptation methods converged at equivalent difficulty in single-task scenarios, with no significant differences in subjective workload or game performance. However, in the dual-task conditions, the model successfully pushed users to higher difficulty levels without performance penalties or increased frustration, highlighting a user tendency to underestimate capacity under high cognitive load. Findings indicate that machine learning models may provide more objective cognitive capacity assessments than self-directed approaches, mitigating subjective performance biases and enabling more effective training by pushing users beyond subjective comfort zones toward physiologically-determined optimal challenge levels.

Figures (13)

• Figure 1: The flowchart depicting the key features and relationship of the experiments used in this study.
• Figure 2: In-game view at different difficulty levels including mental arithmetic menu (middle right) and perceived difficulty selection menu (top right).
• Figure 3: The flowchart depicting the experimental flow of the initial design used for data collection (experiment 1 on the left) and the adaptive design used to evaluate the real-time deployment of the model (experiment 2 on the right).
• Figure 4: Diagram of the LSTM classifier architecture.
• Figure 5: Confusion matrix, ROC curves and feature importance rating for the offline (pre-deployment) model.