Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics

TL;DR

This study explores how neuromodulation, a building block of learning in biological systems, can help address catastrophic forgetting and enhance the robustness of ANNs in continual learning, paving the way for ANNs with greater flexibility, robustness, and adaptability.

Abstract

Continuous, adaptive learning, the ability to adapt to the environment and keep improving performance, is a hallmark of natural intelligence. Biological organisms excel in acquiring, transferring, and retaining knowledge while adapting to volatile environments, making them a source of inspiration for artificial neural networks (ANNs). This study explores how neuromodulation, a building block of learning in biological systems, can help address catastrophic forgetting and enhance the robustness of ANNs in continual learning. Driven by neuromodulators including dopamine (DA), acetylcholine (ACh), serotonin (5-HT) and noradrenaline (NA), neuromodulatory processes in the brain operate at multiple scales, facilitating dynamic responses to environmental changes through mechanisms ranging from local synaptic plasticity to global network-wide adaptability. Importantly, the relationship between neuromodulators and their interplay in modulating sensory and cognitive processes is more complex than previously expected, demonstrating a "many-to-one" neuromodulator-to-task mapping. To inspire neuromodulation-aware learning rules, we highlight (i) how multi-neuromodulatory interactions enrich single-neuromodulator-driven learning, (ii) the impact of neuromodulators across multiple spatio-temporal scales, and correspondingly, (iii) strategies for approximating and integrating neuromodulated learning processes in ANNs. To illustrate these principles, we present a conceptual study to showcase how neuromodulation-inspired mechanisms, such as DA-driven reward processing and NA-based cognitive flexibility, can enhance ANN performance in a Go/No-Go task. Though multi-scale neuromodulation, we aim to bridge the gap between biological and artificial learning, paving the way for ANNs with greater flexibility, robustness, and adaptability.

Figures (3)

• Figure 3: The role of neuromodulators in species of increasing cognitive complexity. Although neuromodulatory systems are broadly conserved across species, they exhibit a progressive departure from stereotyped functions as neural and cognitive complexity increases. From nematodes to humans, neuromodulators support a finer functional specification going from basic sensorimotor and homeostatic processes to higher-order functions such as motivation, attention, memory, affect regulation, and adaptive decision making, which in complex brains give rise to social cognition, abstract reasoning, and creative abilities. This expansion illustrates how a shared neuromodulatory basis may enable more advanced and complex behavior across increasing nervous system complexity.
• Figure 4: The complex relationship between neuromodulators. Modulatory ($\bullet-$): One neuromodulator modulates the release, transmission and/or functional output of the other neuromodulator. Convergent (gradient color bar): Neuromodulators exhibit overlapping, yet sometimes distinctive effects on sensory and cognitive processes. Opponent ($\rightarrow\leftarrow$): Neuromodulators exert opposing effects, or one suppresses the activity of the other. DA: dopamine; 5-HT: serotonin; ACh: acetylcholine; NA: noradrenaline.
• Figure 5: Contingency adaptation in spiking neural networks during a Go--No-Go task.A) Experimental paradigm. B) Neuromodulatory actor--critic architecture, comprising a predictive network and a critic network, with DA and NA signals regulating learning and gain. C) Input structure before and after the set shift. Two visual stimuli and two auditory stimuli are encoded as sparse population activity. D) Task performance over time for DA-only learning (red), DA-only "lucky" trials that successfully adapt by chance (orange), and DA + NA co-modulated learning (green). The vertical line represents the set shift. E) The LC-tone transient at the set shift. F) Conditioned action entropy given the go cue, $H(A \mid \mathrm{goCue})$, quantifying exploratory behavior. DA: dopamine; NA: noradrenaline; LC: locus coeruleus.