Moderating Role of Presence in EEG Responses to Visuo-haptic Prediction Error in Virtual Reality

TL;DR

The paper addresses how to continuously quantify presence in VR by linking moment-to-moment sensorimotor prediction errors to neural activity. Using a within-subject design that contrasts visual-only and visuo-haptic immersion with occasional glitches, the study combines EEG with presence questionnaires to identify ERP and oscillatory signatures of prediction error, localized to ACC and PCC. Findings show robust PEN-like fronto-central ERP components and parietal negativities to mismatches, with PCC alpha suppression selectively enhanced under high-immersion mismatches, supporting a hierarchical predictive-coding account where frontal monitoring is immersion-invariant while posterior multisensory integration scales with sensory precision. These results advance understanding of presence in VR and point to neural markers that could enable neuroadaptive VR systems, though inter-individual presence effects did not reliably moderate neural responses in this setup.

Abstract

Virtual reality (VR) can create compelling experiences that evoke presence, the sense of ``being there.'' However, problems in rendering can create sensorimotor disruptions that undermine presence and task performance. Presence is typically assessed with post-hoc questionnaires, but their coarse temporal resolution limits insight into how sensorimotor disruptions shape user experience. Here, we combined questionnaires with electroencephalography (EEG) to identify neural markers of presence-affecting prediction error in immersive VR. Twenty-five participants performed a grasp-and-place task under two levels of immersion (visual-only vs.~visuo-haptic). Occasional oddball-like sensorimotor disruptions introduced premature feedback to elicit prediction errors. Overall, higher immersion enhanced self-presence but not physical presence, while accuracy and speed improved over time irrespective of immersion. At the neural level, sensorimotor disruptions elicited robust event-related potential effects at FCz and Pz, accompanied by increases in frontal midline $θ$ and posterior $α$ suppression. Through source analyses localized to anterior- and posterior cingulate cortex (ACC/PCC) we found that PCC $α$ activity showed heightened sensitivity to disruptions exclusively in visuo-haptic immersion. Exploratory moderation analyses by presence scores revealed no consistent patterns. Together, these results suggest that higher immersion amplifies both the benefits and costs of sensorimotor coherence.

Figures (7)

• Figure 1: Experimental setup. EEG was recorded using a 64-channel BrainProducts amplifier system. Haptic feedback was delivered through the SenseGlove Nova interface. The virtual environment was presented via the HTC VIVE Pro Eye headset. Hand position and movement were tracked using a VIVE Tracker.
• Figure 2: Task sequence. Each trial began by pressing a start button, after which one or more objects appeared following a random interval (1–2 s). Participants reached toward the object, grasped it with a precision grip, transported it to the target location, placed it, and released it. In mismatch trials, premature visual and/or haptic feedback was presented during the reaching phase. The task was self-paced and lasted on average 10 s per trial.
• Figure 3: Behavioral results. a: Self-presence ratings, b: mean placement error (cm), and c: trial duration by immersion level and experiment phase. Error bars indicate the standard error of the mean.
• Figure 4: ERP results. a: Estimated ERP effects at FCz and Pz showing model-derived beta coefficients for the baseline and experimental factors (congruency, immersion, and their interaction). b: Scalp topographies of the experimental effects at representative latencies (100–300 ms). Enlarged electrodes indicate clusters with significant effects.
• Figure 5: Moderation by presence scores. a: Moderation of model-derived beta coefficients at FCz and Pz by $\Delta$ presence scores. b: Scalp topographies of moderation effects at representative latencies (100–300 ms).