SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning

TL;DR

SynBrain addresses the fundamental one-to-many nature of visual-to-fMRI mappings by modeling neural responses as a semantic-conditioned probability distribution. It combines BrainVAE, a probabilistic, semantically aligned latent space, with the S2N Mapper, a one-step Transformer that projects visual semantics into this latent space, enabling fast and stable fMRI synthesis. Empirical results show superior subject-specific encoding, strong few-shot cross-subject adaptation, and effective use of synthetic fMRI for augmenting fMRI-to-image decoding, while revealing interpretable cross-trial and cross-subject functional patterns. This probabilistic framework offers a biologically grounded digital twin of visual cortical processing and has broad implications for brain decoding and data-efficient neural interfaces.

Abstract

Deciphering how visual stimuli are transformed into cortical responses is a fundamental challenge in computational neuroscience. This visual-to-neural mapping is inherently a one-to-many relationship, as identical visual inputs reliably evoke variable hemodynamic responses across trials, contexts, and subjects. However, existing deterministic methods struggle to simultaneously model this biological variability while capturing the underlying functional consistency that encodes stimulus information. To address these limitations, we propose SynBrain, a generative framework that simulates the transformation from visual semantics to neural responses in a probabilistic and biologically interpretable manner. SynBrain introduces two key components: (i) BrainVAE models neural representations as continuous probability distributions via probabilistic learning while maintaining functional consistency through visual semantic constraints; (ii) A Semantic-to-Neural Mapper acts as a semantic transmission pathway, projecting visual semantics into the neural response manifold to facilitate high-fidelity fMRI synthesis. Experimental results demonstrate that SynBrain surpasses state-of-the-art methods in subject-specific visual-to-fMRI encoding performance. Furthermore, SynBrain adapts efficiently to new subjects with few-shot data and synthesizes high-quality fMRI signals that are effective in improving data-limited fMRI-to-image decoding performance. Beyond that, SynBrain reveals functional consistency across trials and subjects, with synthesized signals capturing interpretable patterns shaped by biological neural variability. Our code is available at https://github.com/MichaelMaiii/SynBrain.

Figures (9)

• Figure 1: Overview of SynBrain for subject-adaptive visual-to-fMRI synthesis and downstream decoding applications. SynBrain is trained on full fMRI recordings from a source subject and adapted to novel subjects using limited data. It generates semantically consistent neural responses that support brain functional analysis and enhance downstream decoding through synthetic data augmentation.
• Figure 2: Overview of the SynBrain framework.Stage 1: BrainVAE models the probabilistic distribution of fMRI responses conditioned on CLIP visual embeddings $z_{CLIP}$; Stage 2: S2N Mapper learns to map $z_{CLIP}$ into the latent space of BrainVAE; Stage 3: At inference, the frozen S2N Mapper performs a one-step mapping from $z_{\mathrm{CLIP}}$ to the BrainVAE latent space for visual-to-fMRI synthesis. Synthesized fMRI could be further visualized via a pretrained fMRI-to-image generator.
• Figure 3: Architecture and performance comparison of MLP-based baselines and our proposed BrainVAE. Left: Architecture comparisons; Right: Validation performance comparisons.
• Figure 4: Visual-to-fMRI synthesis results of SynBrain and fMRI-to-image visualizations.
• Figure 5: Cross-trial and cross-subject brain functional consistency visualization. Left: Comparisons of activation maps between different fMRI trials and our synthesized fMRI evoked by the same stimuli. Right: Comparisons of activation maps between Sub2 (Full-data, 40h) and Sub1$\to$Sub2 (Few-Shot, 1h) evoked by representative categories of visual stimuli.