Decoding Dynamic Visual Experience from Calcium Imaging via Cell-Pattern-Aware SSL

TL;DR

The paper tackles the challenge of self-supervised learning on heterogeneous neural calcium imaging data by proposing POYO-SSL, which pretrains exclusively on statistically regular (predictable) neurons identified via low skewness and kurtosis, and then finetunes on the unpredictable population. This data-diet strategy yields significant gains on downstream tasks, including high-fidelity movie reconstruction (SSIM $\approx 0.593$) and improved drifting-grating classification, while enabling stable scaling with model size. The approach is underpinned by theoretical analyses showing a smooth loss landscape and higher information content in predictable neurons, and by comprehensive representational diagnostics demonstrating more efficient latent spaces. Together, these results establish a principled, biologically grounded framework for scalable neural decoding and offer a path toward foundation models of neural dynamics.

Abstract

Self-supervised learning (SSL) holds a great deal of promise for applications in neuroscience, due to the lack of large-scale, consistently labeled neural datasets. However, most neural datasets contain heterogeneous populations that mix stable, predictable cells with highly stochastic, stimulus-contingent ones, which has made it hard to identify consistent activity patterns during SSL. As a result, self-supervised pretraining has yet to show clear signs of benefits from scale on neural data. Here, we present a novel approach to self-supervised pretraining, POYO-SSL that exploits the heterogeneity of neural data to improve pre-training and achieve benefits of scale. Specifically, in POYO-SSL we pretrain only on predictable (statistically regular) neurons-identified on the pretraining split via simple higher-order statistics (skewness and kurtosis)-then we fine-tune on the unpredictable population for downstream tasks. On the Allen Brain Observatory dataset, this strategy yields approximately 12-13% relative gains over from-scratch training and exhibits smooth, monotonic scaling with model size. In contrast, existing state-of-the-art baselines plateau or destabilize as model size increases. By making predictability an explicit metric for crafting the data diet, POYO-SSL turns heterogeneity from a liability into an asset, providing a robust, biologically grounded recipe for scalable neural decoding and a path toward foundation models of neural dynamics.

Figures (11)

• Figure 1: Overall Framework of POYO-SSL.(a) Pretraining strategy using predictable calcium traces with masked reconstruction learning (50% masking on temporal dimension). (b) Task-specific finetuning with unpredictable traces using either skip-connection UNet decoder (complex tasks) or original POYO+ decoder (simple tasks). (c) Skip-Connection UNet Decoder architecture replacing traditional encoder skip connections with neural embedding projections.
• Figure 2: Loss Landscape Topology Reveals a Dichotomy in Optimization Difficulty. Reconstruction loss landscapes for predictable and unpredictable neurons, projected onto their first two principal components (PCs). (a) The landscape from predictable neurons is smooth and convex-like, with data points (red dots) concentrated in a single basin, indicating a well-posed optimization problem. (b) In contrast, the landscape from unpredictable neurons is rugged and non-convex, characterized by numerous local minima, which presents a challenging, ill-posed problem.
• Figure 3: End-to-end neural-to-vision decoding. (a-e) depict movie frames presented to the mouse, (f-j) depict reconstructed frames. Our model captures subtle frame-to-frame variations (red boxes), demonstrating true reconstruction rather than frame memorization.
• Figure 4: POYO-SSL learns a more efficient and structured latent manifold.(a-c) t-SNE visualization of latent spaces from our POYO-SSL model (a), a reverse SSL model (b), and a from-scratch model (c). Point color reflects the temporal progression of frames. (d) Quantitative comparison of the Intrinsic Dimension (ID) for each model.
• Figure 5: Pre-training with predictable neurons is crucial for effective model scaling.(a) Test SSIM performance versus model size for different pre-training strategies. Only the model pretrained exclusively on predictable neurons (red) demonstrates robust, positive scaling with model capacity. In contrast, training from scratch (gray) or including unpredictable neurons in pre-training (cyan, orange) leads to flat or erratic scaling. (b) Corresponding validation loss during pre-training on the predictable set, showing a general downward trend that indicates successful learning.