SCDM: Unified Representation Learning for EEG-to-fNIRS Cross-Modal Generation in MI-BCIs

TL;DR

This work tackles the practical barrier of acquiring hybrid EEG–fNIRS data for MI‑BCIs by introducing SCDM, a diffusion‑based cross‑modal generator that synthesizes fNIRS from EEG signals. The framework couples a spatial cross-modal generation (SCG) module with a multi‑scale temporal representation (MTR) module within a U‑Net backbone, guided by a forward diffusion process and a Wasserstein distribution alignment objective. Empirical results show synthetic fNIRS closely resembles real signals and yields joint EEG+synthetic‑fNIRS classification that rivals or exceeds EEG+real fNIRS, with synthetic fNIRS preserving spatial and temporal features such as hemodynamic responses and scalp topographies. The findings suggest a practical pathway to obtain abundant hybrid EEG–fNIRS data, broadening MI‑BCI applicability, and point to future work on generalization across datasets and potential enhancements with 3D spatio‑temporal convolutions.

Abstract

Hybrid motor imagery brain-computer interfaces (MI-BCIs), which integrate both electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals, outperform those based solely on EEG. However, simultaneously recording EEG and fNIRS signals is highly challenging due to the difficulty of colocating both types of sensors on the same scalp surface. This physical constraint complicates the acquisition of high-quality hybrid signals, thereby limiting the widespread application of hybrid MI-BCIs. To facilitate the acquisition of hybrid EEG-fNIRS signals, this study proposes the spatio-temporal controlled diffusion model (SCDM) as a framework for cross-modal generation from EEG to fNIRS. The model utilizes two core modules, the spatial cross-modal generation (SCG) module and the multi-scale temporal representation (MTR) module, which adaptively learn the respective latent temporal and spatial representations of both signals in a unified representation space. The SCG module further maps EEG representations to fNIRS representations by leveraging their spatial relationships. Experimental results show high similarity between synthetic and real fNIRS signals. The joint classification performance of EEG and synthetic fNIRS signals is comparable to or even better than that of EEG with real fNIRS signals. Furthermore, the synthetic signals exhibit similar spatio-temporal features to real signals while preserving spatial relationships with EEG signals. Experimental results suggest that the SCDM may represent a promising paradigm for the acquisition of hybrid EEG-fNIRS signals in MI-BCI systems.

Figures (5)

• Figure 1: Framework diagram of SCDM, and its core modules SCG and MTR.
• Figure 2: Radar chart of the classification results.
• Figure 3: The distribution of the most correlated EEG channels with real and synthetic HbR signals. The section concerning real HbR also illustrates the sensors' spatial arrangement and recording site distribution of EEG-fNIRS.
• Figure 4: Comparison of hemodynamic response curves between real and synthetic HbR/ HbO under LMI and RMI tasks.
• Figure 5: Comparison of scalp topography between real and synthetic HbR/ HbO under LMI and RMI tasks.