Auto Detecting Cognitive Events Using Machine Learning on Pupillary Data

TL;DR

The trade-offs between generalization and specialization, model behavior when encountering unseen stimulus onset times, structural variances among cognitive tasks, factors influencing model predictions, and real-time simulation are discussed.

Abstract

Assessing cognitive workload is crucial for human performance as it affects information processing, decision making, and task execution. Pupil size is a valuable indicator of cognitive workload, reflecting changes in attention and arousal governed by the autonomic nervous system. Cognitive events are closely linked to cognitive workload as they activate mental processes and trigger cognitive responses. This study explores the potential of using machine learning to automatically detect cognitive events experienced using individuals. We framed the problem as a binary classification task, focusing on detecting stimulus onset across four cognitive tasks using CNN models and 1-second pupillary data. The results, measured by Matthew's correlation coefficient, ranged from 0.47 to 0.80, depending on the cognitive task. This paper discusses the trade-offs between generalization and specialization, model behavior when encountering unseen stimulus onset times, structural variances among cognitive tasks, factors influencing model predictions, and real-time simulation. These findings highlight the potential of machine learning techniques in detecting cognitive events based on pupil and eye movement responses, contributing to advancements in personalized learning and optimizing neurocognitive workload management.

Figures (7)

• Figure 1: Mean Pupil Changed after ST for each task. Mean Change with Error Bars (1 Standard Deviation) following the start time for all 57 participants. The plots for DPT, PVT, MA, and VWM represented the respective task averages, while the "All-tasks" plot indicated the overall average across all datasets. The X-axis depicted time relative to the ST, while the Y-axis represented the difference between PD at the ST and PD at corresponding times.
• Figure 2: Mean Gaze Position Shift after ST in 57 participants for each task. The graphs for DPT, PVT, MA, and VWM showed task-specific means, while the "All-tasks" plot illustrated the average across all tasks. The X-axis denoted time relative to the ST, and the Y-axis displayed the variation between PD at the ST and PD at corresponding time points.
• Figure 3: Analysis of Individual Participant Pupil Diameters. The top panel presented the distribution of median pupil diameters across 57 participants. Participant count was represented on the left Y-axis via bars, while the probability density function was depicted on the right Y-axis (yellow line). The X-axis represented median Pupil Diameter in millimeters. The middle and lower panels displayed the mean changes in pupil diameter over time in response to a stimulus for each participant. The X-axis indicated time relative to the ST, while the Y-axis represented the variation between pupil diameter at the ST and pupil diameter at corresponding time points. The middle panel showed the average of three tasks: MA, PVT and VWM, while the lower panel focused exclusively on the DPT.
• Figure 4: Sample Generation Process. The plot illustrated the process of sample generation. The black dotted line represented the Stimulus Time (ST). For every ST, the generation process produced three samples: two "0" samples (yellow) and one "1" sample (red). Each sample contained one-second of data.
• Figure 5: Confusion matrices of five models trained and tested on the same dataset. Four task-specific models ("DPT", "MA", "PVT", "VWM") trained and tested separately on their respective datasets, while the "All-task" model trained and tested on the entire dataset. Each matrix corresponded to a single model. The panels included the "All-task" (upper left), "DPT" (upper middle), "MA" (upper right), "PVT" (lower left), and "VWM" (lower middle). The Y-axis represented the true label, while the X-axis represented the predictions. The model names and four metrics, namely accuracy, F1 score, AUC, and MCC, were displayed above their respective matrices.