Accelerating Neural ODEs Using Model Order Reduction

TL;DR

It is shown that mathematical model order reduction (MOR) methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces and can facilitate efficient, dynamical system-driven DL in resource-constrained applications.

Abstract

Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time-series naturally and incorporate knowledge of physical systems into deep learning models. However, the practical applications of Neural ODEs are limited due to long inference times, because the outputs of the embedded ODE layers are computed numerically with differential equation solvers that can be computationally demanding. Here we show that mathematical model order reduction methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and interpolation operations as layers of the neural network. We validate our approach by comparing it to neuron pruning and SVD-based weight truncation methods from the literature in image and time-series classification tasks. The methods are evaluated by acceleration versus accuracy when adjusting the level of compression. On this spectrum, we achieve a favourable balance over existing methods by using model order reduction when compressing a convolutional Neural ODE. In compressing a recurrent Neural ODE, SVD-based weight truncation yields good performance. Based on our results, our integration of model order reduction with Neural ODEs can facilitate efficient, dynamical system-driven deep learning in resource-constrained applications.

Figures (7)

• Figure 1: A plain discrete feedforward network on the left and a residual network on the right. The defining feature of residual networks is that the output from an earlier layer skips layers and is added directly to a later layer.
• Figure 2: A Neural ODE on the left with the discretized differential equation block in orange color. A POD-DEIM reduced network on the right illustrates that the ODE block is evaluated in a low-dimensional subspace, with linear transformations around the ODE block. The networks have equal inputs and approximately equal outputs.
• Figure 3: Metrics for the convolutional Neural ODE. Left: Singular values POD (purple) and DEIM (green) snapshot matrices and singular values of the weight matrix of the ODE block (yellow). Right: histogram of APoZ scores, with the first 100 DEIM indices indicated in green color.
• Figure 4: Performance of reduced convolutional Neural ODEs on the MNIST dataset, relative to the original model. Left plot shows results without fine-tuning, and right plot with three epochs of tuning. The x-axis shows achieved acceleration, while the y-axis shows reduced model top-1 accuracy divided by full model accuracy. Reduced model dimension decreases to the direction of improving speedup. Absolute performance can be seen in Table A.\ref{['table:conv']}.
• Figure 5: Metrics of the ODE-RNN. Left: Singular values of POD (purple) and DEIM (green) snapshot matrices and singular values of the weight matrix of the ODE block (yellow). Right: histogram of APoZ scores, with the first 50 DEIM indices indicated in green color.