Diffusion of Neuromodulators for Temporal Credit Assignment

TL;DR

This work shows how diffusion-based modulation can provide a plausible mechanism for credit assignment in sparsely connected neural circuits, and applies it to recurrent spiking neural networks with sparse feedback connectivity.

Abstract

Biological learning achieves temporal credit assignment despite sparse and imprecise feedback, often relying on neuromodulatory signals acting over space and time. Here, we introduce a learning mechanism in which error information diffuses locally through the network, similar to volume transmission of neuromodulators. This distributed modulation allows neurons to learn even in the absence of direct feedback, using the local concentration of the diffusing credit signal. Applied to recurrent spiking neural networks with sparse feedback connectivity, diffusive credit signaling improves learning across three benchmark tasks. Using eligibility propagation as a baseline learning mechanism, we show how diffusion-based modulation can provide a plausible mechanism for credit assignment in sparsely connected neural circuits.

Figures (2)

• Figure 1: Network architecture and local connectivity pattern. (a.) A spatially embedded RSNN with local connectivity receives inputs from an external input layer. The activities of the RSSN are processed by a readout layer to generate the output. The RSNN is sparsely connected to the input and readout layers. (b.) LIF and ALIF cells are randomly placed in an equally spaced 2D grid. Neurons are connected preferentially to nearby neurons. Only a few cells are connected to the readout layer and receive direct neuromodulatory feedback. Once upon arrival, the neuromodulatory signal diffuses to neighboring neurons.
• Figure 2: Diffusion of neuromodulatory signal improves performance of e-prop when RSNN is sparsely connected to readout. Upper row: task schematic. Lower: learning curves for BPTT, e-prop, and e-prop with diffusion. Solid lines indicate the average value across 20 runs, and shaded areas the standard error of the mean. (a.) Pattern generation, where for visualization purposes we plotted the normalized MSE $\text{ nMSE} = {\sum_{k,t} \left( y^*_{k,t} - y_{k,t} \right)^2}/{\sum_{k,t} \left( y^*_{k,t} \right)^2}$ for zero-mean target $y^*_{k,t}$. (b.) Delayed match to sample. (c.) Cue accumulation. The displayed results are for the post-release neuromodulatory signal decay rate of $k=0.75$. However, the results are qualitatively similar for $k \in \{ 0.25, 0.5, 0.9\}$, clearly outperforming the version without diffusion.