Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models

Changli Tang, Shurui Li, Junliang Wang, Qinfan Xiao, Zhonghao Zhai, Lei Bai, Yu Qiao, Bowen Zhou, Wen Wu, Yuanning Li, Chao Zhang

TL;DR

NOBEL takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding and integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI.

Abstract

Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce \textbf{NOBEL}, a \textbf{n}euro-\textbf{o}mni-modal \textbf{b}rain-\textbf{e}ncoding \textbf{l}arge language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.

One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models

TL;DR

NOBEL takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding and integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI.

Abstract

Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce \textbf{NOBEL}, a \textbf{n}euro-\textbf{o}mni-modal \textbf{b}rain-\textbf{e}ncoding \textbf{l}arge language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.
Paper Structure (17 sections, 5 equations, 2 figures, 5 tables)

This paper contains 17 sections, 5 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation Models for EEG and MEG Signals
  4. Foundation Models for fMRI Signals
  5. Methodology
  6. Model Architecture
  7. Training Pipeline
  8. Experimental Setup
  9. Data Specifications
  10. Model and Training Specifications
  11. Experimental Results
  12. Results on EEG, MEG, or voxel-based fMRI Tasks
  13. Results on stimuli-aware beta-based fMRI tasks
  14. Synergistic Omni-Modal Decoding
  15. Conclusion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Overview of the NOBEL architecture. The model integrates heterogeneous brain signals and external stimuli into an LLM, features dual-path fMRI encoders for disentangling metabolic features (raw voxels vs. beta weights) and a unified EEG/MEG encoder for electromagnetic dynamics. All brain modalities are projected into the LLM's input space via modality-specific 3-Layer MLP aligners. The system supports flexible input combinations and is optimized via LoRA for instruction-following tasks.
  • Figure 2: Examples of NOBEL decoding results on NSD. Each case displays the original visual stimulus presented to the subject alongside the reference caption and the model's predicted caption. These demonstrate that NOBEL can predict semantically accurate image descriptions directly from brain activity.