Dopamine-driven synaptic credit assignment in neural networks

TL;DR

This work tackles the synaptic Credit Assignment Problem by replacing traditional backpropagation with a derivative-free Weight Perturbation optimizer, Dopamine, driven by Reward Prediction Error to adapt learning rates. The core update is $\Delta \theta = - \frac{\eta}{\sigma^2} \mathcal{R} \xi$, with simultaneous perturbations across layers and an adaptive $\eta$ derived from a moving-average of $\mathcal{R}$. Empirically, Dopamine accelerates convergence and reduces memory/computation relative to gradient-based methods while achieving competitive performance on XOR classification and chaotic time-series forecasting, and it improves biological plausibility through local updates and neuromodulatory signals, aided by spectral WP for RNN stability. Limitations include hyperparameter sensitivity and residual error from perturbations, suggesting avenues for theoretical grounding and adaptive-noise strategies.

Abstract

Solving the synaptic Credit Assignment Problem(CAP) is central to learning in both biological and artificial neural systems. Finding an optimal solution for synaptic CAP means setting the synaptic weights that assign credit to each neuron for influencing the final output and behavior of neural networks or animals. Gradient-based methods solve this problem in artificial neural networks using back-propagation, however, not in the most efficient way. For instance, back-propagation requires a chain of top-down gradient computations. This leads to an expensive optimization process in terms of computing power and memory linked with well-known weight transport and update locking problems. To address these shortcomings, we take a NeuroAI approach and draw inspiration from neural Reinforcement Learning to develop a derivative-free optimizer for training neural networks, Dopamine. Dopamine is developed for Weight Perturbation (WP) learning that exploits stochastic updating of weights towards optima. It achieves this by minimizing the regret, a form of Reward Prediction Error (RPE) between the expected outcome from the perturbed model and the actual outcome from the unperturbed model. We use this RPE to adjust the learning rate in the network (i.e., creating an adaptive learning rate strategy, similar to the role of dopamine in the brain). We tested the Dopamine optimizer for training multi-layered perceptrons for XOR tasks, and recurrent neural networks for chaotic time series forecasting. Dopamine-trained models demonstrate accelerated convergence and outperform standard WP, and give comparable performance to gradient-based algorithms, while consuming significantly less computation and memory. Overall, the Dopamine optimizer not only finds robust solutions and comparable performance to the state-of-the-art Machine Learning optimizers but is also neurobiologically more plausible.

Figures (12)

• Figure 1: Multi-layered feed-forward module $g$. $g$ denotes nonlinearity and $x$ denotes the input to the network. $y(t)=g(x_t);l \in [1,L]$; $L=2$. Solid arrows indicate forward data flow with unperturbed parameters, while dotted arrows represent flow with perturbed parameters. Red dotted arrows illustrate the parallel, layerwise parameter update step.
• Figure 2: Recurrent core module $g_i$ unrolled over time. $h$ represents the recurrent state. $y_t,h_t=g(x_t,h_{t-1});$$g_i=g \ \ \forall_i;\tau=[t,T],T=3$. Solid arrows denote the temporal data flow through neuron $g_i$ and the red dotted arrows illustrate the time-independent parameter update step.
• Figure 3: RPE $\mathcal{R}$ modulates learning rate $\eta$ while solving XOR-classification problem. Left: Dopamine-1: Learning rate follows the RPE($\mathcal{R}$) as observed in biological reinforcement learning urbanczik2014learningcoddington2023mesolimbic. Right: Dopamine-2: Exponentially decays the learning rate towards $\mathcal{R}$, thereby avoiding abrupt parameter updates that might arise during the initial training stages. See \ref{['fig:betas_etas']} for more details on the influence of moving average coefficients $\beta_s$ and $\beta_\eta$ and also the initial value of the auxiliary variable $s_0$ and the learning rate $\eta_0$. See hyperparameters details at \ref{['tab:xor_hyperparam_ablation']} for Dopamine 1c (Left) and Dopamine 2b (Right).
• Figure 4: Impact of hyperparameters in Dopamine algorithms. a) Training curves for Dopamine 1 (dotted) and Dopamine 2(solid) for different moving average coefficient values ($\beta_{s},\beta_{\eta}$). See \ref{['tab:xor_hyperparam_ablation']} for detailed hyperparameter settings. b) Influence of initial value of auxiliary variable $s_0$ and initial learning rate $\eta_0$ on the final loss for an RNN trained on Rossler attractor data. c) Impact of moving average coefficients on the performance of the RNN trained on Rossler attractor data. Detailed hyperparameter settings are provided in \ref{['tab:rossler_hyperparam_ablation']}.
• Figure 5: XOR classification task: Loss Landscape computed by random direction method li2018visualizing using 1000 steps in [-20, 20] for $\theta_0$ and [-15, 15] for $\theta_1$. The resulting loss (color) surface discretized into 10 contour regions. Red dot indicates the coordinates of the optimal parameters on the loss landscape. See \ref{['tab:xor_hyperparam']} for hyperparameter details and \ref{['fig:loss_landscape']} for other benchmarks.