Overcoming Output Dimension Collapse: When Sparsity Enables Zero-shot Brain-to-Image Reconstruction at Small Data Scales

TL;DR

This paper tackles the challenge of zero-shot brain-to-image reconstruction under severe data scarcity by comparing naive and sparse linear translators within a Translator–Generator pipeline. It shows that naive regression inherently suffers output dimension collapse (ODC) at small data scales, confining predictions to the span of training outputs and creating irreducible latent-feature error. By contrast, sparse brain-to-feature mappings can extend predictions beyond this subspace, and the authors derive analytic expressions for prediction error in a student–teacher setting, highlighting when sparsity yields gains. Empirical analysis on real fMRI data (Deeprecon) confirms ODC as a substantial contributor to prediction error but also demonstrates that gains from sparsity are context-dependent, improved by lower noise and appropriately structured feature representations. Collectively, the work provides quantitative diagnostics to diagnose ODC and actionable guidelines for translator design and measurement strategies to enable more robust zero-shot brain decoding.

Abstract

Advances in brain-to-image reconstruction are enabling us to externalize the subjective visual experiences encoded in the brain as images. A key challenge in this task is data scarcity: a translator that maps brain activity to latent image features is trained on a limited number of brain-image pairs, making the translator a bottleneck for zero-shot reconstruction beyond the training stimuli. In this paper, we provide a theoretical analysis of two translator designs widely used in recent reconstruction pipelines: naive multivariate linear regression and sparse multivariate linear regression. We define the data scale as the ratio of the number of training samples to the latent feature dimensionality and characterize the behavior of each model across data scales. We first show that the naive linear regression model, which uses a shared set of input variables for all outputs, suffers from ``output dimension collapse'' at small data scales, restricting generalization beyond the training data. We then analyze sparse linear regression models in a student--teacher framework and derive expressions for the prediction error in terms of data scale and other sparsity-related parameters. Our analysis clarifies when variable selection can reduce prediction error at small data scales by exploiting the sparsity of the brain-to-feature mapping. Our findings provide quantitative guidelines for diagnosing output dimension collapse and for designing effective translators and feature representations for zero-shot reconstruction.

Figures (11)

• Figure 1: Translator--Generator pipeline. The translator converts brain activity into latent features, and the generator transforms latent features into reconstructed images. A naive multivariate linear regression model and a sparse multivariate linear regression model are commonly used as translators. A naive model uses a shared set of input variables to predict all output dimensions, and a sparse model performs variable selection for each output dimension.
• Figure 2: Output dimension collapse. (A) The predictions become restricted to a low-dimensional subspace determined by the training outputs. (B) The best prediction within the training feature subspace provides the lower bound for prediction error in naive multivariate linear regression.
• Figure 3: The best predictions for different training data sizes. (A) The best prediction error $\frac{1}{d_\mathrm{out}}\|\hat{\bm{y}}_{\mathrm{te}}^{\mathrm{opt}} - \bm{y}_{\mathrm{te}}\|^2$ for each test sample across training data sizes 1200, 600, 300, and 150. The vertical axis represents the error of the best prediction, and the horizontal axis represents the sample index sorted by the error at $n =$150. (B) Representative reconstructions from the best predictions for different training data sizes. From top: ground truth, reconstructed images generated from the true features $\mathcal{G}(\bm{y}_{\mathrm{te}})$, the best predictions $\mathcal{G}(\hat{\bm{y}}_{\mathrm{te}}^{\mathrm{opt}})$ using 1200, 600, 300, and 150 samples. The left four columns show natural images, and the right two columns show artificial shape images.
• Figure 4: Comparison of the best prediction and the brain prediction. (A) Left: Percentage of the best prediction error relative to the brain prediction error $\|\hat{\bm{y}}_{\mathrm{te}}^{\mathrm{opt}} - \bm{y}_{\mathrm{te}}\|^2/\|\hat{\bm{y}}_{\mathrm{te}} - \bm{y}_{\mathrm{te}}\|^2$. Blue for natural images, orange for artificial shapes. The error bars denote the standard deviation across samples. Right: Sample-wise comparison of best prediction error $\frac{1}{d_\mathrm{out}}\|\hat{\bm{y}}_{\mathrm{te}}^{\mathrm{opt}} - \bm{y}_{\mathrm{te}}\|^2$ and brain prediction error $\frac{1}{d_\mathrm{out}}\|\hat{\bm{y}}_{\mathrm{te}} - \bm{y}_{\mathrm{te}}\|^2$. Each dot represents one image, and the dotted line indicates where the two errors are equal. (B) Representative reconstructions generated from the best predictions and brain predictions. From top: ground truth, reconstructions from the true features $\mathcal{G}(\bm{y}_{\mathrm{te}})$, the best predictions $\mathcal{G}(\hat{\bm{y}}_{\mathrm{te}}^{\mathrm{opt}})$, brain predictions $\mathcal{G}(\hat{\bm{y}}_{\mathrm{te}})$. The bottom row shows reconstructions from the original method of shen2019deep. The left four columns show natural images, and the right two columns show artificial shape images.
• Figure 5: Sparse brain-to-feature mapping. (A) If the brain and the latent features exhibit local or selective activity, their mapping should become sparse. (B) Teacher model in our student--teacher framework, reflecting the sparse structure of the brain-to-feature mapping. The smaller the non-zero weight ratio $a$, the sparser the input-output mapping is.

Theorems & Definitions (1)

• proof