Neuronal Synchrony in Complex-Valued Deep Networks

TL;DR

The paper tackles the limitation that standard deep networks lack mechanisms for representing spike-timing information by introducing complex-valued neural units that encode both firing rate and phase. It presents a concrete formulation where outputs depend on both synchrony and magnitude, including a stabilizing term to handle excitation and inhibition, and demonstrates that pretrained real-valued networks can be converted to complex-valued ones to study synchrony-driven phenomena. Through bars, corners, and shape/MNIST datasets, the authors show that neurons can dynamically form synchrony assemblies that bind distributed components and permit phase-based readout of objects. These results suggest that neuronal synchrony could serve as a versatile mechanism for gating information flow and performing dynamic, phase-guided segmentation in deep networks, with substantial implications for learning with temporal structure.

Abstract

Deep learning has recently led to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to cortical circuits. The challenge is to identify which neuronal mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile representations. We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter, we demonstrate the potential of the approach in several simple experiments. Thus, neuronal synchrony could be a flexible mechanism that fulfills multiple functional roles in deep networks.

Figures (7)

• Figure 1: Transmission of rhythmical activity, and corresponding model using complex-valued units. (a) A Hodgkin–Huxley model neuron receives two rhythmic spike trains as input, plus background activity. The inputs are modeled as inhomogeneous Poisson processes modulated by sinusoidal rate functions (left; shown are rates and generated spikes), with identical frequencies but differing phases. The output of the neuron is itself rhythmical (right; plotted is the membrane potential). (b) The neuron's output rate is modulated by the phase difference between the two inputs (rate averaged over 15s runs). (c) We represent the timing of maximal activity of a neuron as the phase of a complex number, corresponding to a direction in the complex plane. The firing rate is the magnitude of that complex number. Also shown is the color coding used to indicate phase throughout this paper (thus, figures should be viewed in color). (d) The outputs of the input neurons are scaled by synaptic weights (numbers next to edges) and added in the complex plane. The phase of the resulting complex input determines the phase of the output neuron. The activation function $f$ is applied to the magnitude of the input to compute the output magnitude. Together, this models the influence of synchronous neuronal firing on a postsynaptic neuron. (e) Output magnitude as function of phase difference of two inputs. With a second term added to a neuron's input, out-of-phase excitation never cancels out completely (see main text for details; curves are for $w_{1}=w_{2}>0$, $|z_{1}|=|z_{2}|$). Compare to 1b.
• Figure 1: Additional results. (a) Samples generated from a restricted Boltzmann machine trained on the bars problem. The generated images consist mostly of full-length bars. The individual receptive fields in the hidden layer were constrained to image regions of smaller extent than the bars. Thus, bars were necessarily represented in a distributed fashion. (b) - (e) Additional examples of synchronized visible units for the various datasets. The magnitudes of the visible units were set according to the binary input images (not used in training), the phases were determined by input from the hidden units. See also supplementary videos (http://arxiv.org/abs/1312.6115), and the main text for details.
• Figure 2: Gating of interactions. Out-of-phase input, when combined with a stronger input, is weakened. In this example, with $\Delta\phi=\pi$ and as long as $|\mathbf{w}_{1}\cdot\mathbf{z}_{1}|>|w_{2}z_{2}|$, effective input from the neuron to the right is zero, for any input strength (classic and synchrony terms contributions cancel, bottom panel). Hence, neuronal groups with different phases (gradually) decouple.
• Figure 3: Binding by synchrony in shallow, distributed representations. (a) Each image of our version of the bars problem contained 6 vertical and 6 horizontal bars at random positions. (b) A restricted Boltzmann machine was trained on bars images and then converted to a complex-valued network. The magnitudes of the visible units were clamped according to the input image (bottom left), whereas the hidden units and phases of the visible units were activated freely. After 100 iterations, units representing the various bars were found to have synchronized (right; the phases are color-coded for units that are active; black means a unit is off). The neurons synchronized even though receptive fields of the hidden units were constrained to be smaller than the bars. Thus, binding by synchrony could make the 'independent components' of sensory data explicit in distributed representation, in particular when no single neuron can possibly represent a component (a full-length bar) on its own. (c) Histogram of the unit phases in the visible layer for the example shown in b.
• Figure 4: Binding by synchrony in a deep network. (a) Each image contained four corners arranged in a square shape, and four randomly positioned corners. (b) The four corners arranged in a square were usually found to synchronize. The synchronization of the corresponding hidden units is also clearly visible in the hidden layers. The receptive field sizes in the first hidden layer were too small for a hidden unit to 'see' more than individual corners. Hence, the synchronization of the neurons representing the square in the fist hidden and visible layers was due to feedback from higher layers (the topmost hidden layer had global connectivity).