Model-brain comparison using inter-animal transforms

TL;DR

The paper addresses how to robustly compare artificial neural network activations to brain responses across subject variability. It introduces the Inter-Animal Transform Class (IATC), the strictest set of mappings needed to align neural responses between subjects, and advocates bidirectional mappings between models and brains. Across simulated model populations, mouse electrophysiology, and human fMRI data, IATC achieves high predictivity and strong mechanism specificity, with activation nonlinearities shaping the mapping via a zippering dynamic. The work provides convergent evidence for topographical deep neural networks (TDANNs) as models of the visual system and demonstrates that principled, bidirectional IATC-guided comparisons improve upon previous model-brain assessment approaches.

Abstract

Artificial neural network models have emerged as promising mechanistic models of the brain. However, there is little consensus on the correct method for comparing model activations to brain responses. Drawing on recent work in philosophy of neuroscience, we propose a comparison methodology based on the Inter-Animal Transform Class (IATC) - the strictest set of functions needed to accurately map neural responses between subjects in an animal population. Using the IATC, we can map bidirectionally between a candidate model's responses and brain data, assessing how well the model can masquerade as a typical subject using the same kinds of transforms needed to map across real subjects. We identify the IATC in three settings: a simulated population of neural network models, a population of mouse subjects, and a population of human subjects. We find that the IATC resolves detailed aspects of the neural mechanism, such as the non-linear activation function. Most importantly, we find that the IATC enables accurate predictions of neural activity while also achieving high specificity in mechanism identification, evidenced by its ability to separate response patterns from different brain areas while strongly aligning same-brain-area responses between subjects. In other words, the IATC is a proof-by-existence that there is no inherent tradeoff between the neural engineering goal of high model-brain predictivity and the neuroscientific goal of identifying mechanistically accurate brain models. Using IATC-guided transforms, we obtain new evidence in favor of topographical deep neural networks (TDANNs) as models of the visual system. Overall, the IATC enables principled model-brain comparisons, contextualizing previous findings about the predictive success of deep learning models of the brain, while improving upon previous approaches to model-brain comparison.

Figures (10)

• Figure 1: Model-brain comparison using inter-animal transforms. The Inter-Animal Transform Class (IATC) is the strictest set of functions required to map responses accurately between subjects in a population for a given brain area. We propose to use the empirically identified IATC to map bidirectionally between a candidate model's responses and brain responses in order to assess whether the model can masquerade as a typical animal subject. The arrows represent individual functions in the IATC. These functions may not in general be invertible, so the arrows are not symmetric. Nonetheless, bidirectional mapping emerges from the fact that it is possible to map responses between any pair of subjects in the natural population using the IATC, and thus we map in both directions when comparing a model to the brain, just as we would when comparing two actual brains. This allows us to separate between models that do match the brain under the IATC and those that do not.
• Figure 2: Evaluating candidate transform classes for specificity and hierarchy. (A) It is desirable for a mapping to achieve high specificity -- simultaneously achieving within-area identifiability and between-area separability when mapping responses between animals. Each dot is a response profile for a particular subject and brain area. The schematic barplots represent likely outcomes for model separation when comparing models to brains. (B) To capture more graded relationships beyond specificity, we also look at the correlation between dissimilarity scores and distances in a known hierarchy.
• Figure 3: Assessing same-area similarity in the model population reveals a "zippering" effect caused by the model activation function. (A) We attempt to identify the IATC for a model population by first assessing within-area similarity when mapping between differently seeded model subjects. (B) The Zippering Effect: Post-non-linearity activations are only moderately similar between subjects up to linear transform, especially for the intermediate layers. However, pre-non-linearity activations are highly similar at all layers. At every layer, the non-linear activation function causes responses between different subjects to diverge somewhat up to linear transform, only for responses to converge again at the next layer's pre-activations, before diverging and converging again. (C) Post-non-linearity responses can be thought of as corresponding to firing rates, while pre-non-linearity responses can be thought of as corresponding to EPSPs (excitatory post synaptic potentials). (D) The Zippering Transform: The Zippering Effect provides a clue as to how to construct a better transform class - namely, by accounting for details of the mechanism (the non-linear activation function). Step 1 inverts the non-linearity to recover the pre-non-linearity activations of one subject, step 2 applies a fitted linear transform to predict the pre-non-linearity activations of the other subject, and step 3 re-applies the non-linearity to predict post-non-linearity activations.
• Figure 4: Predictivity and specificity for a spectrum of candidate transform classes on a simulated model population. (A) Same-layer predictivity when mapping responses between model subjects. (B) Specificity and hierarchy correlation for different transform classes. Error bars are 95% CIs. (C) A scatterplot comparing predictivity and specificity across transform classes. While the exact shape of the Pareto frontier for predictivity and specificity is unknown, we identify a bounded region (shaded blue) that contains at least one Pareto-optimal point. The diagonal of this region represents the maximum possible distance from our best IATC candidate (Exact Zippering) to the Pareto frontier. (D) Multidimensional scaling (MDS) plots to visualize dissimilarity scores, when mapping response profiles between all layers and all subjects. Each dot is a response profile for a particular subject and model layer. Distances between dots are optimized to match the dissimilarities using a particular comparison method.
• Figure 5: Assessing candidate transform classes on a mouse population. (A and B) Mapping responses from pooled mouse subjects to a held-out subject in order to evaluate same-area predictivity (A) and specificity (B). (C) We map five layers from four models to the mouse data: the ReLU-based AlexNet model of mouse cortex nayebi2022mouse, our noisy softplus version of that model, ResNet, and VGG16. We evaluate the mean absolute difference between models (averaged over model pairs and model layers) in terms of assessed brain similarity (using the noise-corrected Pearson correlation or RSA score).