Functional connectivity guided deep neural network for decoding high-level visual imagery

TL;DR

The paper addresses the challenge of decoding high-level visual imagery from non-invasive EEG signals for brain-computer interfaces aimed at intuitive control of robotic devices. It introduces the Functional Connectivity-guided Deep Neural Network (FCDN), which fuses PLV-based functional connectivity with a CNN–image transformer (DeiT) and frequency-specific processing to capture both spatial and temporal dynamics across delta, theta, and alpha bands. Key contributions include a novel FC layer that emphasizes spatial interactions, comprehensive evaluations (offline, pseudo-online, LOSO), and ablation analyses demonstrating the importance of occipital involvement and FC guidance. The findings show competitive to superior decoding performance, including subject-independent gains, suggesting potential for practical EEG-based robotic arm control and broader BCI applications.

Abstract

This study introduces a pioneering approach in brain-computer interface (BCI) technology, featuring our novel concept of high-level visual imagery for non-invasive electroencephalography (EEG)-based communication. High-level visual imagery, as proposed in our work, involves the user engaging in the mental visualization of complex upper limb movements. This innovative approach significantly enhances the BCI system, facilitating the extension of its applications to more sophisticated tasks such as EEG-based robotic arm control. By leveraging this advanced form of visual imagery, our study opens new horizons for intricate and intuitive mind-controlled interfaces. We developed an advanced deep learning architecture that integrates functional connectivity metrics with a convolutional neural network-image transformer. This framework is adept at decoding subtle user intentions, addressing the spatial variability in high-level visual tasks, and effectively translating these into precise commands for robotic arm control. Our comprehensive offline and pseudo-online evaluations demonstrate the framework's efficacy in real-time applications, including the nuanced control of robotic arms. The robustness of our approach is further validated through leave-one-subject-out cross-validation, marking a significant step towards versatile, subject-independent BCI applications. This research highlights the transformative impact of advanced visual imagery and deep learning in enhancing the usability and adaptability of BCI systems, particularly in robotic arm manipulation.

Figures (8)

• Figure 1: Experimental protocols for high-level visual imagery from electroencephalography (EEG) signals. (a) Experimental environment for acquiring high-level visual imagery data. (b) Experimental paradigm in a single trial and representation of visual cues according to each task. (c) The type of stimuli given to users in the experiment: picking up a cell phone (class-1), pouring water (class-2), opening a door (class-3), and eating food (class-4).
• Figure 2: The connections between the channels that have functional connectivity scores above 0.9 when the high-performance group (Sub14, Sub09, and Sub04) and low-performance group (Sub01, Sub08, and Sub13) performed a high-level visual imagery task. Functional connectivity was assessed in the delta and alpha frequency ranges. In the delta band, connections are mainly represented in the prefrontal area as shown in the high-performance group. On the other hand, the low-performance group's connections are irregular. In the alpha band, the third row, the high-performance group's connections are mainly represented in the occipital area. However, the low-performance group's connections show irregular tendencies.
• Figure 3: The overview of the proposed architecture begins with the segmentation of the raw electroencephalogram (EEG) into delta waves, theta waves, and alpha waves. In the functional connectivity layer, each of these EEG wave types is then multiplied channel-wise using a functional connectivity score measured by the phase-locking value. The modified EEG data that emphasize spatial information is used as the input data for the deep learning architecture. We extracted temporal information using a network consisting of a convolutional block, followed by spatial information via a transformer block.
• Figure 4: In the functional connectivity layer, the raw electroencephalogram (EEG) data undergo transformation through the phase locking value (PLV). Each channel's connectivity score, derived via the PLV, is then normalized. The subsequent integration with the original EEG accentuates spatial details, producing an EEG output enriched with heightened spatial significance.
• Figure 5: The power spectral changes in each selected channel. The yellow box in the graph indicates the frequency range containing a significant peak. The peaks were observed in channels (a) Fz and (b) Oz related to visual imagery. The time-frequency analysis (c) presents the distinct characteristics of high-level visual imagery. Prominent features are evident in the alpha band at the Fz and Oz locations, which are known to be associated with visual imagery.