Deep Oscillatory Neural Network

TL;DR

The DONN network exhibits emergent phenomena such as feature and temporal binding during image classification, a key characteristic of biological visual processing, and exhibit STDP (Spike Timing Dependent Plasticity) kernel when trained using Hebbain learning, which enhances the interpretability of internal representations.

Abstract

We propose a novel, brain-inspired deep neural network model known as the Deep Oscillatory Neural Network (DONN). Deep neural networks like the Recurrent Neural Networks indeed possess sequence processing capabilities but the internal states of the network are not designed to exhibit brain-like oscillatory activity. With this motivation, the DONN is designed to have oscillatory internal dynamics. Neurons of the DONN are either nonlinear neural oscillators or traditional neurons with sigmoidal or ReLU activation. The neural oscillator used in the model is the Hopf oscillator, with the dynamics described in the complex domain. Input can be presented to the neural oscillator in three possible modes. The sigmoid and ReLU neurons also use complex-valued extensions. All the weight stages are also complex-valued. Training follows the general principle of weight change by minimizing the output error and therefore has an overall resemblance to complex backpropagation. A generalization of DONN to convolutional networks known as the Oscillatory Convolutional Neural Network is also proposed. The two proposed oscillatory networks are applied to a variety of benchmark problems in signal and image/video processing. The performance of the proposed models is either comparable or superior to published results on the same data sets.

Figures (13)

• Figure 1: The Oscillatory Neural Network is generally composed of alternating static and oscillatory layers.
• Figure 2: Mechanisms through which the input, $z_{in}(t)$ can be presented to the Oscillator unit
• Figure 3: Driven Behaviour of a Supercritical Hopf Oscillator Under Strong Forcing. Steady-state amplitude and relative phase as a function of frequency difference $(\mu = 1, \beta_1 = -100, \beta_2 = 0, F = 0.2).$
• Figure 4: Convolutional Oscillatory block: feature maps obtained after convolutional operation on input image at time $t$ is given as input $I(t)$ in one-to-one fashion to oscillatory block of dimension equal to that of feature maps to obtain spatio-temporal feature maps
• Figure 5: The performance of the model for signal generation task to generate the four classes reliably.