Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors

TL;DR

This work addresses the challenge of predicting dynamic neural activity by jointly modeling external video stimuli and internal brain states. It introduces a probabilistic latent-state model that infers a stimulus-independent latent z from a subset of neurons and integrates it with a video-driven core to predict time-varying neural responses via a Zero-Inflated Gamma distribution. Across SENSORIUM mouse V1 data, the latent model outperforms video-only approaches in likelihood and shows that learned latent factors strongly correlate with behavior and exhibit topographic cortical organization, even without behavioral or anatomical inputs during training. The results demonstrate that unsupervised latent factor learning can reveal meaningful structure linking sensory processing and behavior, with potential to generalize to unseen neurons using cortical coordinates and to other core architectures.

Abstract

The neural activity in the visual processing is influenced by both external stimuli and internal brain states. Ideally, a neural predictive model should account for both of them. Currently, there are no dynamic encoding models that explicitly model a latent state and the entire neuronal response distribution. We address this gap by proposing a probabilistic model that predicts the joint distribution of the neuronal responses from video stimuli and stimulus-independent latent factors. After training and testing our model on mouse V1 neuronal responses, we find that it outperforms video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned on responses from other neurons. Furthermore, we find that the learned latent factors strongly correlate with mouse behavior and that they exhibit patterns related to the neurons' position on the visual cortex, although the model was trained without behavior and cortical coordinates. Our findings demonstrate that unsupervised learning of latent factors from population responses can reveal biologically meaningful structure that bridges sensory processing and behavior, without requiring explicit behavioral annotations during training.

Figures (10)

• Figure 1: A. Problem statement: an animal watches the same video, but different brain states lead to different neuronal responses. B. Proposed solution: infer the brain state with a probabilistic latent-variable model that predicts marginal Zero-Inflated-Gamma (ZIG) distributions for each neuron.
• Figure 2: A. A 3D convolutional neural network-based core extracts features of the video input $\mathbf{x}$. The read-out learns the spatial position (purple dot). The deterministic Poisson baseline predicts the mean response via a per-neuron affine function from the learned feature vector $\mathbf{w}^c$. For the ZIG baseline, we double the dimension of $\mathbf{w}^c$ to $\mathbf{w}^{(q,\theta)}$ to predict response-distribution parameters $q,\theta$ of a ZIG distribution.
• Figure 3: A Average prediction–response correlation across models with varying latent dimensions. B Average conditioned correlation for low- vs. high-dimensional latent models, if different portions of the neuron population are given. C Log-likelihood vs. average norm of latent feature vectors $\mathbf{w}_{i}^{(q)},\mathbf{w}_{i}^{(\theta)}$ across latent dimensions. Error bars: SEM over 3 seeds; some points omitted to save training cost.
• Figure 4: Canonical correlation of behavior and latent for each mouse. The CCA analysis was done with 5-fold cross-validation in an 80/20 split. The error bars indicate the standard error of the mean of cross-validation.
• Figure 5: Color gradient maps of the latent feature vectors $\textbf{w}_i^{(q)}$ along the first three Singular Dimensions for mouse 9. All recorded neurons are located within a $600\mu m \times 600\mu m$ square in the cortex. Their depth differs at most 200 $\mu m$. The first three columns display the color maps for a model with freely learned feature vectors, trained without any knowledge of cortical positions. Columns 1-2 use only neurons from a specific depth for both SVD and visualization; Column 3 includes all neurons for both. Column 4 shows a model predicting latent features from cortical positions. Rows show the top three singular dimensions in descending order.