Anticipation Before Action: EEG-Based Implicit Intent Detection for Adaptive Gaze Interaction in Mixed Reality

TL;DR

This study tackles the Midas Touch problem in MR by examining SPN, an EEG marker that reflects anticipatory processing, as an implicit indicator of user intention during gaze-based interaction. Using a within-subject 2×2 design (Intention: Select vs Observe; Feedback: With vs Without) across three MR scenarios, the authors show SPN is robustly elicited and modulated by both internal goals and external feedback, with the strongest SPN observed in Observe-No Feedback conditions. They also demonstrate that deep learning models can decode user intention from anticipatory EEG signals in a person-dependent setting (up to 97% accuracy) and provide interpretability analyses (LIME) confirming SPN-based features drive predictions. These results support a shift from motor-preparation interpretations of SPN toward an uncertainty-monitoring view, enabling adaptive MR interfaces that adjust dwell times and confirmations based on neural state, potentially reducing Midas Touch and enabling personalized interaction. The work offers practical pathways for integrations with multimodal signals (e.g., pupil size, EDA) to build robust, intention-aware MR systems with real-time decoding capabilities.

Abstract

Mixed Reality (MR) interfaces increasingly rely on gaze for interaction , yet distinguishing visual attention from intentional action remains difficult, leading to the Midas Touch problem. Existing solutions require explicit confirmations, while brain-computer interfaces may provide an implicit marker of intention using Stimulus-Preceding Negativity (SPN). We investigated how Intention (Select vs. Observe) and Feedback (With vs. Without) modulate SPN during gaze-based MR interactions. During realistic selection tasks, we acquired EEG and eye-tracking data from 28 participants. SPN was robustly elicited and sensitive to both factors: observation without feedback produced the strongest amplitudes, while intention to select and expectation of feedback reduced activity, suggesting SPN reflects anticipatory uncertainty rather than motor preparation. Complementary decoding with deep learning models achieved reliable person-dependent classification of user intention, with accuracies ranging from 75% to 97% across participants. These findings identify SPN as an implicit marker for building intention-aware MR interfaces that mitigate the Midas Touch.

Figures (11)

• Figure 1: Experimental scenarios illustrated with MR interfaces screenshots. Columns represent the four tasks (Training, Document, App Launcher, Video), and rows show the interface states Before Selection (top) and After Selection (bottom). During training, participants interacted with neutral geometric shapes, while in the main scenarios they performed gaze-based interactions in everyday MR contexts such as editing text, launching applications, browsing files, and controlling video playback. Feedback was presented immediately after gaze-based selections.
• Figure 2: Overview of the experimental Procedure. Participants first completed a short training phase in which neutral geometric shapes (triangle, circle, square) were used instead of real icons. The main experiment followed, where each participant performed all four conditions (Intent: Observe / Select × Feedback: With / No) across three everyday MR scenarios (App Launcher, Document, Video). The order of blocks was counterbalanced using a balanced Latin Williams square design. Refer to \ref{['sec:procedure']} for a complete description of the procedure and to \ref{['sec:task']} for a complete descriptions of the scenarios in the task.
• Figure 3: Schematic of the four trial types (Select vs. Observe, with and without Feedback). All trials began with Instruction (2000 ms), a Fixation Cross (1000 ms + variable jitter of 250, 500, or 750 ms), and a Dwell Time (750 ms). The subsequent phases differed by condition. In the Select – No Feedback condition , the dwell was followed directly by a UI Action (2000 ms) and then the ISI (1000 ms). In the Select – Feedback condition , the dwell was followed by a 500 ms Icon Feedback signal, then a UI Action (2000 ms) and the ISI (1000 ms). In the Observe – No Feedback condition , the dwell was followed immediately by the ISI (1000 ms) without any system response. In the Observe – Feedback condition , the dwell was followed by a 500 ms Icon Feedback signal and then the ISI (1000 ms). Continuous lines indicate that a given phase (Icon Feedback or UI Action) was present in that condition, while dotted lines indicate that the phase was absent. Each condition comprised 90 trials per participant. In Observe trials, NO system action occurred. the interface remained static (No Feedback) or displayed only brief confirmatory feedback (With Feedback). Only Select trials triggered consequential UI changes.
• Figure 4: Trial structure across Intent and Feedback conditions. The timeline illustrates the sequential phases of a single trial across four experimental conditions: Select (with and without feedback) and Observe (with and without feedback). All trials began with a 2-second instruction screen, followed by a fixation cross with a jittered duration (1000 + 250 / 500 / 750 ms). Participants then fixated on a UI element for 750 ms to select or observe it. In Select trials, a UI interaction was triggered for 3000 ms, coherent with the intention to Select. In Observe trials, no interaction followed the gaze dwell. When feedback was enabled, it was presented immediately after dwell time for 500 ms, preceding the UI phase. Finally, all trials concluded with a 1-second inter-stimulus interval (ISI) to allow attentional reset. For visualization purposes, the figure aligns all conditions to the same trial duration (8500 ms).
• Figure 5: Grand-average ERP waveforms and SPN scalp topographies across experimental conditions. The ERP plot displays the grand-average activity at the posterior ROI ( O1, Oz, O2, Iz, PO7, PO3, POz, PO4, PO8, P5, P6, P7, P8, P9, and P10) with shaded areas indicating the 95% confidence intervals. The pre-stimulus baseline window (–1000 to –750 ms) is shown in purple, and the SPN window (–750 to 0 ms) is shaded in beige; the vertical dashed line indicates stimulus onset (0 ms). The scalp maps below illustrate mean voltage distributions during the SPN interval for each condition. Observe--No Feedback elicited the strongest anticipatory negativity, whereas both the intention to select and the presence of feedback reduced the SPN magnitude.