Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

TL;DR

The authors establish a principled link between deep residual networks and discretizations of ordinary differential equations, using this lens to design the linear multi-step LM-architecture that can be applied to ResNet-like networks. They show LM-ResNet and LM-ResNeXt achieve higher accuracy with similar or fewer parameters on CIFAR and ImageNet, and provide a modified-equation explanation for the gains. They also connect stochastic noise injections during training to stochastic dynamic systems, offering a stochastic control perspective and demonstrating improvements with stochastic depth on LM-ResNet. The work suggests that leveraging numerical analysis and stochastic control can guide efficient, scalable neural-network design.

Abstract

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

Figures (5)

• Figure 1: Schematics of network architectures.
• Figure 2: LM-architecture is an efficient structure that enables ResNet to achieve same level of accuracy with only half of the parameters on CIFAR10.
• Figure 3: Training and testing curves of ResNext29 (16x64d, pre-act) and and LM-ResNet29 (16x64d, pre-act) on CIFAR100, which shows that the LM-ResNeXt can achieve higher accuracy than ResNeXt.
• Figure 4: The trained parameters $\{k_n\}$ of LM-ResNet on CIFAR100.
• Figure 5: Training and testing curves of ResNet56 (pre-act) and and LM-ResNet56 (pre-act) on CIFAR10 with stochastic depth.