On the dynamics of convolutional recurrent neural networks near their critical point

TL;DR

This work presents analytical solutions for the steady states when the network is forced with a single oscillation and when a background value creates a steady state of ongoing activity, and derives the relationships shaping the value of the temporal decay and spatial propagation length as a function of this background value.

Abstract

We examine the dynamical properties of a single-layer convolutional recurrent network with a smooth sigmoidal activation function, for small values of the inputs and when the convolution kernel is unitary, so all eigenvalues lie exactly at the unit circle. Such networks have a variety of hallmark properties: the outputs depend on the inputs via compressive nonlinearities such as cubic roots, and both the timescales of relaxation and the length-scales of signal propagation depend sensitively on the inputs as power laws, both diverging as the input to 0. The basic dynamical mechanism is that inputs to the network generate ongoing activity, which in turn controls how additional inputs or signals propagate spatially or attenuate in time. We present analytical solutions for the steady states when the network is forced with a single oscillation and when a background value creates a steady state of ongoing activity, and derive the relationships shaping the value of the temporal decay and spatial propagation length as a function of this background value.

Figures (7)

• Figure 1: Schematic of the map Eq \ref{['eq:model']}. For any given element in the layer, all the inputs are added together; the inputs from other elements in the layer is given by the convolution, and the external input is added element-wise. Finally the activation function is applied, and the result becomes the values of the layer for the next time-step.
• Figure 2: Choose as a basis the “ reshape” operation from an $E\times E$ square to a vector of length $E^{2}$$(0\le i<E,0\le j<E)\ \to\ k\equiv i+Ej\ <E^{2}$ and a convolution which for every point in the lattice adds up the first neighbors with coefficients $y_{ij}=ax_{ij}+bx_{i+1,j}+cx_{i-1,j}+dx_{ij+1}+ex_{ij-1}\qquad\forall ij$. This operation maps the kernel onto a sparse array with lots of repetitive diagonal structures and off-diagonal stuff for boundary conditions. Any matrix property invariant under change of basis is a property of the kernel too. For example, transposition -> central symmetry.
• Figure 3: Fixed point and slope at the fixed point. In these diagrams, the fixed point lies at the intersection of the curve with the diagonal; as the input displaces the curve upwards, the fixed point moves to the right. Left, with a very small input (0.001) representing a nearly invisible displacement of the curve with respect to the diagonal, the fixed point moves right by $\left(2*0.001\right)^{1/3}=0.12549$, and the slope at the fixed point gives a relaxation time of 42 iterations. Right, a much larger input of 0.11 gives rise to a displacement of 0.55, with a relaxation time of 1.8 iterations.
• Figure 4: Fixed point and slope at the fixed point, as a function of input.
• Figure 5: The eigenvalues of $K$ are resonance frequencies of the system, in that forcing at exactly that frequency causes the system to give a disproportionately large response. Resonances of the system show compressive nonlinearities at the resonances, linear behavior away from the resonances, and broadening of the responses as the forcing amplitude increases to match both regimes to each other. Shown are the amplitude of the responses to 5 different intensities separated by $\sqrt{10}$ from each other. Black dots are the result of numerical simulation (N=1024, compleetly random $K$, $2^{24}$ iterations, 1024 frequencies, 5 intensities $10^{-(1:5)/2}$). Red lines are the analytical calculation of Eq. \ref{['eq:nonperturb']}. The blue lines are the eigenvalues of the kernel. Bottom panel: detail showing the widening as the amplitude of the input increases.