Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

TL;DR

The paper tackles how video dynamics are encoded in the brain by performing a large-scale comparison of video understanding models against visual cortex fMRI data, using Mini-Algonauts and BOLD Moments datasets. It introduces a dynamics-aware neural encoding framework that integrates layer-wise features from pre-trained models with ROI-based voxel predictions, and further augments this with inter- and intra-region connectivity priors learned from the data. Key findings show that video models, especially two-stream and multiscale transformers like MViT, align better with brain responses than image models, with convolutional networks excelling in early visual areas, and fully supervised models outperforming self-supervised ones. The paper also demonstrates that incorporating connectivity priors and encoding dynamics yields significant encoding gains, revealing directional influences between visual regions and the importance of motion information for brain–model alignment. These results advance our understanding of neural encoding and offer a framework for building brain-aligned, dynamics-aware models.

Abstract

Extensive literature has drawn comparisons between recordings of biological neurons in the brain and deep neural networks. This comparative analysis aims to advance and interpret deep neural networks and enhance our understanding of biological neural systems. However, previous works did not consider the time aspect and how the encoding of video and dynamics in deep networks relate to the biological neural systems within a large-scale comparison. Towards this end, we propose the first large-scale study focused on comparing video understanding models with respect to the visual cortex recordings using video stimuli. The study encompasses more than two million regression fits, examining image vs. video understanding, convolutional vs. transformer-based and fully vs. self-supervised models. Additionally, we propose a novel neural encoding scheme to better encode biological neural systems. We provide key insights on how video understanding models predict visual cortex responses; showing video understanding better than image understanding models, convolutional models are better in the early-mid visual cortical regions than transformer based ones except for multiscale transformers, and that two-stream models are better than single stream. Furthermore, we propose a novel neural encoding scheme that is built on top of the best performing video understanding models, while incorporating inter-intra region connectivity across the visual cortex. Our neural encoding leverages the encoded dynamics from video stimuli, through utilizing two-stream networks and multiscale transformers, while taking connectivity priors into consideration. Our results show that merging both intra and inter-region connectivity priors increases the encoding performance over each one of them standalone or no connectivity priors. It also shows the necessity for encoding dynamics to fully benefit from such connectivity priors.

Figures (14)

• Figure 1: Architecture of our dynamics based neural encoding with inter-intra region connectivity priors. Our fully integrated model learns the connectivity from all regions voxels (inter-region) and all voxels in the same region (intra-region). We only show one target region, V1, as an illustration where we use the same mechanism across all regions. Note the arrows thickness, in our visual cortex regions illustration, indicate the degree of connectivity corresponding to the computed in Fig. \ref{['fig:connectivity-c']}.
• Figure 2: Simulated experiments showing regression scores as Pearson's correlation coefficient of image (blue) vs. video (red) model families on four target models; (a) MViT-B, (b) ViT-B, (c) ResNet-50, (d) I3D ResNet-50. We show the regression on the target network output features from their respective blocks, B0-7. Statistical significance is shown at the bottom as 'ns' not significant, '$*, **, ***$' significant with p-values $<0.05, 0.01, 0.001$, resp. It shows higher regression scores for the model family corresponding to the target network, especially in MViT and ViT.
• Figure 3: Real experiments showing regression scores as Pearson's correlation coefficient of model families on brain fMRI data. Comparison between: (a) image vs. video understanding models, (b) convolutional vs. transformer-based models and (c) fully supervised vs. self-supervised models. Statistical significance is shown in the bottom as 'ns' not significant, '$*, **, ***$' significant with p-values $<0.05, 0.01, 0.001$, resp. It shows video understanding models outperform single image ones, fully supervised outperform self supervised ones and convolutional models surpass transformer based ones in early-mid regions.
• Figure 4: Fine-grained analysis of the video understanding models across the nine regions of the human visual cortex showing the Pearson's correlation coefficient as the regression scores. (a) Single stream vs. two stream SlowFast architectures. (b) OmniMAE pre-trained in a self-supervised manner vs. TimeSformer and OmniMAE fine-tuned with full supervision. All models are based on ViT-B and trained on SSV2. (c) Comparison between six video understanding models. Statistical significance is shown on top of bar pairs as 'ns' not significant, '$*, **, ***$' significant with p-values $<0.05, 0.01, 0.001$, resp.
• Figure 5: (a) Comparison of base model accuracies of MViT-B-16x4 and SlowFast and their accuracies after incorporating the intra-region and inter-region voxel connectivity showing the Pearson's correlation coefficient as the regression scores. It shows the superiority of the connectivity based models. (b) Comparison of performance enhancement by incorporating the intra-region and inter-region voxel connectivity together or each of them separately showing the Pearson's correlation coefficient as the regression scores. It confirms on the need for combining both intra- and inter-region connectivity. (c) Average weights per region contributing to the accuracy enhancement of each target visual region, showing the directional learned connectivity in our model.