Beyond Gaze Points: Augmenting Eye Movement with Brainwave Data for Multimodal User Authentication in Extended Reality

TL;DR

This work tackles the user authentication bottleneck in XR by fusing eye movement and brainwave biometrics using consumer-grade sensors. It introduces a synchronized data collection and a Twin Neural Network-based feature extractor, evaluating both score- and feature-level fusion across multiple enrollment strategies on 30 participants performing an interactive dot task. The multimodal system achieves markedly lower equal error rates than any single modality, with $EER$ as low as $0.298\%$ (with pupil diameter) and $0.686\%$ (without), outperforming state-of-the-art baselines and demonstrating practical XR viability. The study further analyzes pupil-diameter contributions, threat resilience, and XR integration usability, outlining a clear path toward seamless, secure, hands-free authentication in real-world XR deployments.

Abstract

Extended Reality (XR) technologies are becoming integral to daily life. However, password-based authentication in XR disrupts immersion due to poor usability, as entering credentials with XR controllers is cumbersome and error-prone. This leads users to choose weaker passwords, compromising security. To improve both usability and security, we introduce a multimodal biometric authentication system that combines eye movements and brainwave patterns using consumer-grade sensors that can be integrated into XR devices. Our prototype, developed and evaluated with 30 participants, achieves an Equal Error Rate (EER) of 0.29%, outperforming eye movement (1.82%) and brainwave (4.92%) modalities alone, as well as state-of-the-art biometric alternatives (EERs between 2.5% and 7%). Furthermore, this system enables seamless authentication through visual stimuli without complex interaction.

Figures (6)

• Figure 1: Synchronized collection of eye-movment and brain activity. Lab Streaming Layer (LSL) receives data from PsychoPy, an eye-tracker, and a brainwave device. Also, PsychoPy and the eye-tracker coordinate to execute focus tasks, updating the display and sending focus timestamps to LSL
• Figure 2: A participant in the experimental setup wearing the Pupil Core eye-tracker and the Emotiv EPOC X brainwave device
• Figure 3: In the verification mode of our biometric recognition system, activity data from Alice is captured using a wearable device. This data undergoes pre-processing before entering the Feature Extraction Module, which condenses it into a 32-feature biometric template. Alice's claimed identity (IDc) is then compared with her pre-registered true identity (IDt), which consists of the biometric template established during the enrollment stage. Based on this comparison, the system's Comparison Module determines the legitimacy of Alice's identity.
• Figure 4: Siamese sub-network architectures (A & B) for eye movement and brainwave feature fusion.
• Figure 5: Density distribution of brainwave similarity scores for legitimate users, human attackers, and random input features (it is similar for eye movement and fusion).