Higher-Order Convolution Improves Neural Predictivity in the Retina

TL;DR

The paper introduces Higher-Order Convolution (HoConv) to embed multiplicative spatiotemporal interactions directly into the convolutional operator, enabling shallow networks to better predict retinal neural responses. By replacing the first layer of standard CNNs with HoConv, the authors demonstrate consistent improvements in neural-response prediction across salamander and mouse retinal ganglion cells, achieving correlations near the retinal reliability ceiling while requiring roughly half the training data. The work further reveals that HoCNN naturally encodes core geometric transformations, particularly scaling, and shows cell-type-specific benefits for motion-sensitive RGCs. Overall, the approach bridges a gap between biological computation and artificial models by introducing biologically-inspired primitives that improve efficiency and mechanistic interpretability without increasing architectural depth.

Abstract

We present a novel approach to neural response prediction that incorporates higher-order operations directly within convolutional neural networks (CNNs). Our model extends traditional 3D CNNs by embedding higher-order operations within the convolutional operator itself, enabling direct modeling of multiplicative interactions between neighboring pixels across space and time. Our model increases the representational power of CNNs without increasing their depth, therefore addressing the architectural disparity between deep artificial networks and the relatively shallow processing hierarchy of biological visual systems. We evaluate our approach on two distinct datasets: salamander retinal ganglion cell (RGC) responses to natural scenes, and a new dataset of mouse RGC responses to controlled geometric transformations. Our higher-order CNN (HoCNN) achieves superior performance while requiring only half the training data compared to standard architectures, demonstrating correlation coefficients up to 0.75 with neural responses (against 0.80$\pm$0.02 retinal reliability). When integrated into state-of-the-art architectures, our approach consistently improves performance across different species and stimulus conditions. Analysis of the learned representations reveals that our network naturally encodes fundamental geometric transformations, particularly scaling parameters that characterize object expansion and contraction. This capability is especially relevant for specific cell types, such as transient OFF-alpha and transient ON cells, which are known to detect looming objects and object motion respectively, and where our model shows marked improvement in response prediction. The correlation coefficients for scaling parameters are more than twice as high in HoCNN (0.72) compared to baseline models (0.32).

Figures (8)

• Figure 1: Comparison of standard convolution (Conv) and higher-order convolution (HoConv) blocks.
• Figure 2: Neural response prediction framework. Electrophysiological RGC responses to visual stimuli are first recorded using multi-electrode arrays. These recordings are then used to train convolutional neural networks to predict (multicell) neural responses from the visual input.
• Figure 3: A. Performance comparison across models and training set sizes for maheswaranathan2023interpreting data. B. Single-cell correlation coefficients of the two best performing models: HoCNNv2 vs maheswaranathan2023interpreting on held-out test data. C. Performance comparison across models for maximum size of training data (90% vs 10% validation). D. Single-cell correlation coefficients of the two considered models: HoCNN vs mcintosh2016deep on held-out test data. E. Single-cell correlation coefficients HoCNN vs mcintosh2016deep, trained on the tOFF-$\alpha$ + tON subset, on held-out test data.
• Figure 4: Sample frames from our test set demonstrating perspective transformations applied to a checkerboard pattern. The transformation parameters are fully defined by the homography matrix $\mathbf{H}$. An anonymized sample of the stimulus is available at https://streamable.com/pvrrob.
• Figure 5: Distributions of predicted values for scaling parameters $H_{11}$ from test data, comparing mcintosh2016deep's (A, B, C) and HoCNN model (D, E, F) against ground truth. A Depicts predicted values (linear regressor) against true value for scaling parameter $H_{11}$ from mcintosh2016deep's model (originally trained to predict all the 40 RGCs) features, correlation is $\rho_{McIntosh} = 0.29$; B Same values as A, but from features extracted from mcintosh2016deep's model trained to predict tOFF-$\alpha$ and tON (11 cells), correlation is $\rho_{McIntosh} = 0.32$; C Same values as A and B, but from features extracted from mcintosh2016deep's model trained to predict control RGCs subset (11 cells), correlation is $\rho_{McIntosh} = 0.32$. D, E, F depict the same results as A, B, C from HoCNN's features: D's $\rho_{HoCNN} = 0.56$; E's $\rho_{HoCNN} = 0.72$ and F's $\rho_{HoCNN} = 0.51$.