Scaling laws for decoding images from brain activity

TL;DR

This work investigates how decoding images from brain activity scales with neuroimaging modality and data amount using a unified, single-trial benchmark across eight public datasets. It combines a fixed image embedding (DINOv2) with deep-learning brain modules (M/EEG and fMRI) and diffusion-based reconstruction to compare linear and nonlinear decoders across EEG, MEG, 3T, and 7T fMRI. The authors find that higher-precision devices yield better decoding when data are matched, but deep learning provides the largest gains for noisier recordings; decoding scales log-linearly with data without a plateau and mainly benefits within-subject data, with limited gains from more subjects. They also emphasize that time, cost, and ethical considerations are crucial when planning scalable brain-to-image decoding experiments, guiding optimal data-collection strategies. Overall, the paper lays out practical scaling laws and a rigorous benchmarking framework to advance non-invasive brain decoding toward real-time, resource-aware applications.

Abstract

Generative AI has recently propelled the decoding of images from brain activity. How do these approaches scale with the amount and type of neural recordings? Here, we systematically compare image decoding from four types of non-invasive devices: electroencephalography (EEG), magnetoencephalography (MEG), high-field functional Magnetic Resonance Imaging (3T fMRI) and ultra-high field (7T) fMRI. For this, we evaluate decoding models on the largest benchmark to date, encompassing 8 public datasets, 84 volunteers, 498 hours of brain recording and 2.3 million brain responses to natural images. Unlike previous work, we focus on single-trial decoding performance to simulate real-time settings. This systematic comparison reveals three main findings. First, the most precise neuroimaging devices tend to yield the best decoding performances, when the size of the training sets are similar. However, the gain enabled by deep learning - in comparison to linear models - is obtained with the noisiest devices. Second, we do not observe any plateau of decoding performance as the amount of training data increases. Rather, decoding performance scales log-linearly with the amount of brain recording. Third, this scaling law primarily depends on the amount of data per subject. However, little decoding gain is observed by increasing the number of subjects. Overall, these findings delineate the path most suitable to scale the decoding of images from non-invasive brain recordings.

Figures (14)

• Figure 1: (A) Brain-to-image decoding and encoding pipeline. In decoding, brain models are trained to predict, from brain activity, the embeddings of the images learned by a pretrained computer vision model. Decoding predictions can then be fed to an image generation model to reconstruct the images. In encoding, models are instead trained to predict brain activity from image embeddings. (B) Our analyses rely on multiple datasets of brain data and image pairs, focusing on four neuroimaging devices: EEG, MEG, 3T fMRI and 7T fMRI. (C) We validate the content of the datasets using encoding models trained to predict each M/EEG channel or fMRI voxel from the presented images, which yield the expected spatial response over the occipital region as measured with Pearson correlation. See \ref{['sec:encoding']} for more details.
• Figure 2: Image decoding analyses show the expected temporal response for all datasets. (Left) Subject-specific stepwise decoding using linear ridge regression as a function of time elapsed since image onset ($t=0$). (Right) Sliding window decoding using deep learning models trained across subjects on 100-ms windows for M/EEG or 1 TR for fMRI. We report the average performance across subjects and show the standard error of the mean with shaded areas or error bars. (Bottom) Peak Pearson correlation obtained in the linear (empty circles) and deep learning (filled circles) analyses, for each subject of each study. Circle size indicates the total recording time available for each subject. See \ref{['fig:stepwise_decoding_matched_trials']} for results in the matched-trials setting.
• Figure 3: Decoding of the image embedding as a function of time (x-axis) and number of test-time repetitions (color) using deep learning models. (A) Sliding window decoding (100 ms for M/EEG; 1 TR for fMRI) and (B) growing window decoding ($t_0$=-0.5 for M/EEG; 0.0 for fMRI) show the expected time-locked response and highlights the consistent improvement obtained by adding test-time image repetitions. Grey areas indicate the interval during which images were shown. (C) Peak sliding window performance for each dataset. Black lines and bars indicate performance obtained when averaging predictions over all repetitions of all subjects for each unique test image. We report the average performance across subjects and show the standard error of the mean with shaded areas or error bars. We use "large" architecture configurations everywhere (\ref{['app:architectures']}) except for Grootswagers2022 for which the "medium" configuration yielded more stable training dynamics. See \ref{['fig:test_trial_averaging_matched_trials']} for results in the matched-trials setting.
• Figure 4: Image retrieval across devices. (A) For each representative dataset of each device, a sample of three stimulus images showing some of the most convincing retrievals obtained with our approach. Ground truth images are shown on the top row. Top-1 retrieved images are shown underneath (top-2 image overlaid on bottom right): single-trial brain responses (second row), subject-averaged predictions (third row) and predictions averaged across all subjects (bottom row). (B) Top-5 retrieval accuracies for each dataset and each test-time averaging strategy, grouped by recording device. See \ref{['fig:best_retrievals_matched']} for the matched-trials setting.
• Figure 5: Image reconstruction across devices. (A) For each study, a sample of 3 stimuli showing some of the most convincing reconstructions obtained with our approach: from single-trial brain signals, and two increasingly large aggregations (averaging embedding predictions at subject-level and at instance-level). (B) The comparison of the three image generation metrics PixCorr (pixel space), AlexNet-2-R (low-level, latent space) and CLIP-Final-R (high-level, latent space) for single-trial decoding and aggregations. As expected, performance increases overall when averaging more than one trial, even at subject-level.