Neural units with time-dependent functionality

Abstract

We show that the time-resolved dynamics of an underdamped harmonic oscillator can be used to do multifunctional computation, performing distinct computations at distinct times within a single dynamical trajectory. We consider the amplitude of an oscillator whose inputs influence its frequency. The activity of the oscillator at fixed time is a nonmonotonic function of its inputs, and so it can solve problems such as XOR that are not linearly separable. The activity of the oscillator at fixed input is a nonmonotonic function of time, and so it is multifunctional in a temporal sense, able to carry out distinct nonlinear computations at distinct times within the same dynamical trajectory. We show that a single oscillator, observed at different times, can act as all of the elementary logic gates and can perform binary addition, the latter usually implemented in hardware using 5 logic gates. We show that a set of $n$ oscillators, observed at different times, can perform an arbitrary number of analog-to-$n$-bit digital conversions. We also show that oscillators can be trained by gradient descent to perform distinct classification tasks at distinct times. Computing with time-dependent functionality can be done in or out of equilibrium, and suggests a way of reducing the number of parameters or devices required to do nonlinear computations.

Figures (4)

• Figure 1: An illustration of the computational power of neural units with nonmonotonic activations, using binary inputs $I_{1,2}=0,1$. (a) A standard perceptron unit, whose output is a monotonic function of a linear sum of its inputs, can solve AND. (b) However, it cannot solve XOR, which is not linearly separable: neither red line (nor any other straight line) separates the two classes. (c) A multilayer perceptron is required to solve XOR: its hidden layers transform the input data $(I_1,I_2) \to (\tilde{I}_1,\tilde{I}_2)$ until a linear decision boundary suffices. (d) Alternatively, XOR can be solved by a neural unit whose output is a nonmonotonic function of its inputs vodenicarevic2017neuralgidon2020dendriticnoel2021biologically, here $f(I_1,I_2)=\sin(\pi(I_1+I_2)/2)$. In this paper we build on this observation to show that the dynamics of an underdamped oscillator, whose output is a nonmonotonic function of its inputs and the elapsed time, can perform distinct nonlinear computations at distinct times.
• Figure 2: (a) The time-resolved dynamics (\ref{['sol']}) of the oscillatory neural unit (\ref{['unit']}), or oscillatron, evaluated for the two distinct values of a binary input $I=0,1$. Depending on the observation time $t$, the unit can perform any of the elementary one-bit operations. (b) The time-resolved dynamics of the same unit, now evaluated for the three distinct values $I=I_1+I_2=0,1,2$ of the binary inputs $(I_1,I_2)$. Depending on the observation time $t$, the unit can function as any of the elementary logic gates. (c) A binary full adder realized by the unit (\ref{['unit']}) with input $I=A+B+C_{\rm in}$. We show time traces $S_I(t)$, Eq. (\ref{['sol']}), for the four possible values of $I$. The sum $\Sigma$ and carry-out $C_{\rm out}$ are given by the output of the unit at times $t_1$ and $t_2$, respectively. Oscillator parameters: $\omega_0=1, \gamma=1/10$.
• Figure 3: A graphical representation of Table \ref{['tab1']}, showing that the $i^{\rm th}$ binary digit $B_i$ of the analog input $I$ can be represented by an oscillator of period $2^i$, as $B_i =\Theta \left[S_i(t_I)\right]$.
• Figure 4: Time-dependent multifunctional classification. (a) An oscillator neuron (\ref{['unit']}) with $N=28^2$ adjustable input weights is shown MNIST digits in classes $0,1,4,5,8$, and 9. Its weights are adjusted by gradient descent to distinguish 0s and 1s at time $t_1$; to distinguish 4s and 5s at time $t_2$; and to distinguish 8s and 9s at time $t_3$. (b) Time traces of the activity of the trained oscillator when shown 10 test-set digits (unseen during training) of each indicated class, and histograms of activity (using all digits of that class in the MNIST test set) at the indicated times. (c) As (b), but for a noninteracting gas of 4 oscillatrons, with the 4$N$ input weights adjusted by gradient descent to perform the same multi-time classification task. All histograms have a common vertical scale. (d) Loss (\ref{['phi']}), scaled by $\phi_0=3 N_{\rm batch}$, as a function of training epoch $n$. (e) Activity histogram of the oscillator gas (when shown 0s and 1s) at a time at which it is not trained to discriminate 0s and 1s. In panels (b), (c), and (e), the value zero is indicated horizontal and vertical grey dashed lines. Oscillator parameters: $\omega_0=1, \gamma=1/10$.