Deep Continuous Networks

TL;DR

The paper addresses the mismatch between conventional CNNs and biological vision by introducing Deep Continuous Networks (DCNs) that couple spatially continuous filters with depthwise continuous dynamics via neural ODEs. Filters are defined as Gaussian derivative SRFs with trainable coefficients $\alpha$ and scale $\sigma$, enabling end-to-end learning of both filter shape and spatial extent, while the network depth is treated as a continuous dimension $t$. Empirically, DCNs achieve competitive CIFAR-10 performance with fewer parameters, exhibit data efficiency in small-data settings, and demonstrate improved reconstruction quality; they also reveal biologically plausible scale distributions and support pattern completion, with potential computational savings through contrast-driven time scaling. The work provides a principled bridge between neuroscience-inspired continuous models and modern CNNs, offering a framework for brain-like receptive-field dynamics, efficient parameterization, and new avenues for neuroscientific investigations and energy-efficient deep learning.

Abstract

CNNs and computational models of biological vision share some fundamental principles, which opened new avenues of research. However, fruitful cross-field research is hampered by conventional CNN architectures being based on spatially and depthwise discrete representations, which cannot accommodate certain aspects of biological complexity such as continuously varying receptive field sizes and dynamics of neuronal responses. Here we propose deep continuous networks (DCNs), which combine spatially continuous filters, with the continuous depth framework of neural ODEs. This allows us to learn the spatial support of the filters during training, as well as model the continuous evolution of feature maps, linking DCNs closely to biological models. We show that DCNs are versatile and highly applicable to standard image classification and reconstruction problems, where they improve parameter and data efficiency, and allow for meta-parametrization. We illustrate the biological plausibility of the scale distributions learned by DCNs and explore their performance in a neuroscientifically inspired pattern completion task. Finally, we investigate an efficient implementation of DCNs by changing input contrast.

Figures (5)

• Figure 1: SRF filters based on N-jet filter approximation. Convolutional filters are defined as the weighted sum of Gaussian derivative basis functions up to order 2, with corresponding scales ${\sigma_1 = 2.28}$ (left) and ${\sigma_2=0.90}$ (right). Our DCN models learn both the coefficients $\alpha$, and the scale $\sigma$ end-to-end during training.
• Figure 2: DCN model architecture with CIFAR-10 input images. Convolutional kernel size $k$ is learned during training. The equations of motion (Eq. \ref{['eq:EoM']}) are solved within ODE blocks.
• Figure 3: (a) Learned $\sigma$ values increase with depth within the network. (b) $\sigma^{ji}$ distributions within the ODE blocks display a positive skew in line with biological observations. (c) CIFAR-10 validation accuracies on the pattern completion task with increasing mask size.
• Figure 4: Pattern completion in the DCN feature maps during classification of masked images. Feature maps in a single channel of ODE block 1 are shown for an example image. We find that the difference $D(t)$ between the feature maps $\textbf{h}^\textrm{im}(t)$ of an intact image and $\textbf{h}^\textrm{im\_masked}(t)$ of a masked image is reduced as $t \rightarrow T$. We also show the mean $\overline{D}(t)$ for 1000 validation images (bottom right), where the shaded area is the standard deviation over different images. Example feature maps from baseline models are provided in Appendix A.5.
• Figure 5: On the CIFAR-10 validation set, DCN s are more robust than baseline ODE-Nets to changes in input contrast $c$ at test time (top). Interestingly, the number of function evaluations (NFEs) in the first ODE block (middle) or the whole DCN network (bottom) can be reduced considerably by modulating $c$.