Enhancing Immersion and Presence in the Metaverse with Over-the-Air Brain-Computer Interface

TL;DR

The paper addresses immersive Metaverse experiences under constrained wireless resources by introducing an over-the-air BCI framework where EEG signals are uploaded to a Wireless Edge Server (WES) to jointly learn user actions and allocate radio/computing resources. It formulates a QoE-maximization problem and proposes a Hybrid Learning Algorithm (dual prediction and resource-control) and a Meta-learning Algorithm to handle neurodiversity across users, validated on real EEG data with low latency and high classification accuracy (up to $84\%$). The approach integrates FoV pre-rendering, BACI-style classification, and actor-critic learning with policy clipping to achieve robust performance, outperforming PPO, VPG, and SVM baselines under varied power, CPU, and user-count conditions. This work demonstrates a scalable, brain-data-driven path to enhance immersion and reduce VR sickness in the Metaverse, with practical implications for edge computing and semantic/broadcast brain communications.

Abstract

This article proposes a novel framework that utilizes an over-the-air Brain-Computer Interface (BCI) to learn Metaverse users' expectations. By interpreting users' brain activities, our framework can optimize physical resources and enhance Quality-of-Experience (QoE) for users. To achieve this, we leverage a Wireless Edge Server (WES) to process electroencephalography (EEG) signals via uplink wireless channels, thus eliminating the computational burden for Metaverse users' devices. As a result, the WES can learn human behaviors, adapt system configurations, and allocate radio resources to tailor personalized user settings. Despite the potential of BCI, the inherent noisy wireless channels and uncertainty of the EEG signals make the related resource allocation and learning problems especially challenging. We formulate the joint learning and resource allocation problem as a mixed integer programming problem. Our solution involves two algorithms: a hybrid learning algorithm and a meta-learning algorithm. The hybrid learning algorithm can effectively find the solution for the formulated problem. Specifically, the meta-learning algorithm can further exploit the neurodiversity of the EEG signals across multiple users, leading to higher classification accuracy. Extensive simulation results with real-world BCI datasets show the effectiveness of our framework with low latency and high EEG signal classification accuracy.

Figures (10)

• Figure 1: Illustration of our proposed over-the-air BCI-enabled Metaverse system. A Wireless Edge Server (WES) runs Metaverse applications, supporting VR experiences for $K$ users. These Metaverse users are equipped with integrated VR-BCI headsets. $K$ Digital Avatars (DAs) are maintained in Metaverse to support real-time recommendations and enhance the user QoE.
• Figure 2: Illustration of the FoV pre-rendering process at the WES: (a) a selected video frame before rendering and (b) after rendering . (c) illustrates the computing usage of the FoV pre-rendering process. At the time step 100-th, the computing resource is allocated to $K=3$ users, depicted by the red color circles pointing at the CPU usage of 0.6, i.e., $60\%$ CPU.
• Figure 3: Example of EEG signals recorded from three different BCI participants responding to the same experimental condition (left figure). The EEG signals are extracted from the same channels, i.e., C3, CP3, C4, CP4, Cz, and CPz, denoted as red circles on the surface of the scalp in a 10-10 international system (right figure). These channels are responsible for hands and feet movement morash2008classifying. The instructions to the participants are placed at the time 0 (marked by the vertical dashed line). The neurodiversity, i.e., subjective differences in the same environment, reflect the differences in amplitudes and phases of the BCI participants.
• Figure 4: Training process for the proposed Hybrid learner at the controller of the WES. The circled numbers denote the corresponding steps as described in Fig. \ref{['fig:system-model']} and Section \ref{['sec:system-model']}.
• Figure 5: (a) Normalized QoE, (b) classification accuracy, and (c) average round-trip VR delay values.