LinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Models

TL;DR

The proposed LinBridge framework, a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models, is presented and fresh evidence about hierarchical nonlinearity distribution in the visual cortex is offered.

Abstract

Neural encoding of artificial neural networks (ANNs) links their computational representations to brain responses, offering insights into how the brain processes information. Current studies mostly use linear encoding models for clarity, even though brain responses are often nonlinear. This has sparked interest in developing nonlinear encoding models that are still interpretable. To address this problem, we propose LinBridge, a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models. LinBridge posits that the nonlinear mapping between ANN representations and neural responses can be factorized into a linear inherent component that approximates the complex nonlinear relationship, and a mapping bias that captures sample-selective nonlinearity. The Jacobian matrix, which reflects output change rates relative to input, enables the analysis of sample-selective mapping in nonlinear models. LinBridge employs a self-supervised learning strategy to extract both the linear inherent component and nonlinear mapping biases from the Jacobian matrices of the test set, allowing it to adapt effectively to various nonlinear encoding models. We validate the LinBridge framework in the scenario of neural visual encoding, using computational visual representations from CLIP-ViT to predict brain activity recorded via functional magnetic resonance imaging (fMRI). Our experimental results demonstrate that: 1) the linear inherent component extracted by LinBridge accurately reflects the complex mappings of nonlinear neural encoding models; 2) the sample-selective mapping bias elucidates the variability of nonlinearity across different levels of the visual processing hierarchy. This study presents a novel tool for interpreting nonlinear neural encoding models and offers fresh evidence about hierarchical nonlinearity distribution in the visual cortex.

Figures (19)

• Figure 1: Comparison of linear and nonlinear encoding models. In linear encoding models, the mapping relationship between the feature space and brain activity space is invariant across input samples. On the contrary, nonlinear encoding models exhibit sample-specific characteristics, resulting in an unstable structure that complicates the interpretation of the underlying relationships.
• Figure 2: Nonlinear Encoding Model and the LinBridge Framework. (a) Image representation extraction and the general neural encoding model structure; (b) LinBridge framework, which includes the computation of $\mathbf{JM}$, the extraction of $\mathbf{JM}_{\text{inherent}}$ based on CNN module, the calculation of $\Delta \mathbf{JM}$, and the implementation of a low-dimensional embedding module.
• Figure 3: Comparison of $R^{2}$ between linear and nonlinear encoding models, showing predictions significantly above chance levels ($P<0.05$, FDR corrected). (a) $R^{2}$ in the linear encoding model; (b) $R^{2}$ in the nonlinear encoding model; (c) The histograms of $R^{2}$ in the whole brain, the primary visual cortex (PVC), the secondary visual cortex (SVC), and the tertiary visual cortex (TVC). Results for other subjects are provided in \ref{['subsec:Comparison_encoders']}.
• Figure 4: Comparison of the linear inherent component extracted by LinBridge to the brain activation predicted by the nonlinear encoding model. (a) The stability of the extracted linear inherent component across different batch sizes. (b) Activation patterns of the linear inherent component extracted by LinBridge. (c) Comparison of the distribution of $R^{2}$ values between the linear inherent component extracted by LinBridge and the original nonlinear encoding models in the whole brain, PVC, SVC and TVC. Results for other subjects can be found in \ref{['subsec:Comparison_inherent']}.
• Figure 5: Distribution of nonlinear visual encoding in the brain. (a) The left and right images show voxel fitting results for the highest and lowest evaluation metrics, respectively; (b) Histogram comparison of evaluation metrics for significantly activated voxels in PVC, SVC, and TVC; (c) Visualization of evaluation metrics for significantly activated voxels across the whole brain; (d) Comparison of probability density functions for evaluation metrics of significantly activated voxels in PVC, SVC, and TVC. The sorted image samples can be found in Figure \ref{['FigureApp_3_Sub2']}, and the results for other subjects can be found in \ref{['subsec:Distribution_nonlinear']}.