Multimodal Brain-Computer Interfaces: AI-powered Decoding Methodologies

TL;DR

The paper surveys AI-powered decoding methodologies for multimodal BCIs, addressing how cross-modality mapping, sequential modeling, and multimodal fusion can improve brain data interpretation across visual, speech, and affective domains. It catalogs algorithmic approaches including cross-modality contrastive learning, generative modeling, and Transformer-based fusion, and discusses how these methods enable mappings, translations, and coherent sequencing between brain signals and external modalities. It also analyzes brain data types, acquisition methods, datasets, and practical challenges such as data heterogeneity, big data requirements, and security/privacy concerns, offering a pathway toward brain foundation models. The study highlights potential societal benefits in healthcare, rehabilitation, and brain-computer interfacing while underscoring remaining obstacles to real-world deployment. Overall, it argues that large-scale, aligned multimodal brain data and foundation-model–driven AI will be pivotal for robust, scalable AI-powered BCIs.

Abstract

Brain-computer interfaces (BCIs) enable direct communication between the brain and external devices. This review highlights the core decoding algorithms that enable multimodal BCIs, including a dissection of the elements, a unified view of diversified approaches, and a comprehensive analysis of the present state of the field. We emphasize algorithmic advancements in cross-modality mapping, sequential modeling, besides classic multi-modality fusion, illustrating how these novel AI approaches enhance decoding of brain data. The current literature of BCI applications on visual, speech, and affective decoding are comprehensively explored. Looking forward, we draw attention on the impact of emerging architectures like multimodal Transformers, and discuss challenges such as brain data heterogeneity and common errors. This review also serves as a bridge in this interdisciplinary field for experts with neuroscience background and experts that study AI, aiming to provide a comprehensive understanding for AI-powered multimodal BCIs.

Figures (9)

• Figure 1: The outline of this review.
• Figure 2: Three types of multimodal BCIs and their representative applications. Reactive BCIs involve brain activities stimulated by designed inputs, such as visual decoding, where perceived images are decoded instance-wise. Active BCIs facilitate user-driven communication in sequential forms, such as speech decoding. Passive BCIs record brain activities using various sensors for tasks like affective BCIs. The core mechanisms involved are very distinct in the aspect of AI decoding algorithms for such three types.
• Figure 3: Zero-shot classification under cross-modality contrastive learning, using visual decoding as an example. (a) Training stage, where feature extractors for image and brain signal pairings are learned in the latent space using cross-modality contrastive learning. (b) Test stage, where novel classes that were not present in the training set require classification. Retrieval could be performed using similarity metrics in the latent space.
• Figure 4: Two types of cross-modality generative modeling, using two modalities of image and brain signal as an example. (a) Joint latent space, where separate encoders project two modalities into a shared latent space, and separate decoders reconstruct the respective inputs; (b) Conditional latent space, where an asymmetrical encoder and decoder projects inputs from a modality to a latent space and then to another modality.
• Figure 5: Three pipelines for decoding brain recordings into speech and their respective sequential modeling components. The majority of works decode brain data into sentences, either through direct sequence-to-sequence neural networks, or first map to discrete entities of language and then reorganize them. An alternative pipeline directly synthesizes speech waveform, through first mapping to mel-spectrogram, then using vocoder for audio generation.