Teaching signal synchronization in deep neural networks with prospective neurons

TL;DR

This work addresses the timing problem in learning within hierarchical neural systems by proposing prospective neurons that predict future inputs to synchronize teaching signals with evolving neural activity. The authors formalize delays as tracking errors to an implicit target trajectory and show that standard leaky integrators cannot perfectly track this target, while prospective dynamics can achieve asymptotic tracking and rapid convergence. They provide a bio-physical implementation using adaptive currents, analyze robustness to time-constant mismatches, and demonstrate that prospective neurons enable effective online learning across multiple learning rules, reward-based control, and memory-enabled networks. The findings suggest a biologically plausible mechanism for mitigating internal delays, with broad implications for online learning in dynamic environments and potential links to existing theoretical frameworks such as NLA, LE, and prediction-correction approaches.

Abstract

Working memory requires the brain to maintain information from the recent past to guide ongoing behavior. Neurons can contribute to this capacity by slowly integrating their inputs over time, creating persistent activity that outlasts the original stimulus. However, when these slowly integrating neurons are organized hierarchically, they introduce cumulative delays that create a fundamental challenge for learning: teaching signals that indicate whether behavior was correct or incorrect arrive out-of-sync with the neural activity they are meant to instruct. Here, we demonstrate that neurons enhanced with an adaptive current can compensate for these delays by responding to external stimuli prospectively -- effectively predicting future inputs to synchronize with them. First, we show that such prospective neurons enable teaching signal synchronization across a range of learning algorithms that propagate error signals through hierarchical networks. Second, we demonstrate that this successfully guides learning in slowly integrating neurons, enabling the formation and retrieval of memories over extended timescales. We support our findings with a mathematical analysis of the prospective coding mechanism and learning experiments on motor control tasks. Together, our results reveal how neural adaptation could solve a critical timing problem and enable efficient learning in dynamic environments.

Figures (5)

• Figure 1: Prospective dynamics can track the target trajectory whereas leaky dynamics cannot. Under the leaky integrator dynamics, the current state $s_t$ is attracted towards the current target $s_t^\ast$. By the time it arrives there, the target will have already moved away, so $s$ is always lagging behind $s^\ast$. Adding a prospective input to the leaky dynamics enables perfect tracking, as the position of $s^\ast$ in the near future is indirectly estimated and will serve as a target for the $s$ dynamics.
• Figure 2: Theoretical predictions on tracking properties of leaky and prospective neurons hold in simulation. A. Empirical tracking error for the leaky dynamics (\ref{['eqn:continuous-time-BP']}) as a function of the neurons time constant $\tau$ times the characteristic angular velocity of the inputs $\omega_0$. This quantity is directly linked to the theoretical bound of Theorem \ref{['thm:problem_flow']}. Each dot corresponds to random $\tau$ and $\omega_0$ values. As predicted by Theorem \ref{['thm:problem_flow']}, the tracking error scales linearly with $\tau\omega_0$ in the worst case, which measures how slow the neurons are compared to the input. Consequently, teaching signals arrive out-of-sync. B. Under the prospective dynamics, the instantaneous empirical tracking error converges exponentially fast to 0, with the neurons time constant $\tau$ modulating the convergence speed, consistently with Theorem \ref{['thm:perfect_tracking']}.
• Figure 3: Better target trajectory tracking translates to greater learning performance. Top row measures the empirical tracking error in the setup of Sections \ref{['sec:theory']} and \ref{['sec:physical_constraints']}, bottom row the test loss in the teacher-student task of Section \ref{['subsec:learning-teacher-student']}. Note that the two setups are similar but have some differences, cf. Methods. A, B. Leaky dynamics (\ref{['eqn:continuous-time-BP']}) require $\tau \rightarrow 0$ to perfectly track the reference trajectory and thus effectively learn (blue). Prospective dynamics (\ref{['eqn:prospective_dynamics']}) can do so for any $\tau$ (red), matching the performance of instantaneous backpropagation (grey). C, D Adaptive dynamics (\ref{['eqn:adaptive']}) track and learn better than leaky dynamics and the effect amplifies as $\tau_a$ gets smaller, relatively to $\tau$. E, F. Time constant mismatches as in (\ref{['eqn:mismatch_tau']}) hinder tracking and thus learning in the prospective dynamics. Yet, for a wide range of mismatches, the learning loss remains significantly lower that the one of leaky dynamics.
• Figure 4: Prospective dynamics support learning in a control task.A. Visual depiction of the inverted-pendulum task, in which the goal is to balance the pole in an upright position. Reward rate is $1$ as long as the pole remains in the "target angles" region and that cart is not too far from the origin. B. The current state, comprising the position $x$ and velocity $\dot{x}$ of the cart, and the angle $\theta$ and angular velocity $\dot{\theta}$ of the pole, is fed to an actor-critic network. This network outputs an action (left or right) and an estimated value representing the expected discounted reward. At each time step, the estimated value is combined with the reward to compute the temporal difference (TD) error, which then serves as a teaching signal for both the actor and the critic. C. The prospective backpropagation algorithm solves the task -- maintaining the pole in the target region for $4$s -- as efficiently as its instantaneous counterpart. Gradually reducing the prospective component of dynamics ($\tau' \rightarrow 0$ with $\tau'$ as in Section \ref{['subsec:time_constant_mismatch']}) diminishes learning capability, eventually preventing learning altogether. Increasing $\tau'$ has similar effects. Small deviations around $\tau' = \tau$ do not significantly affect performance, as seen with the $\tau' = \tau - \tau/10$ and $\tau'= \tau + \tau / 10$ curves. We use $\tau = 100\mathrm{ms}$ and report the rolling average of episode length using a $30$-episode window, averaged across $5$ seeds. Standard deviations (approximately $500$ms) are omitted to avoid visual clutter, with no significant differences observed between methods. See Methods section for implementation details.
• Figure 5: Prospective neurons support learning of memory-storing non-prospective leaky neurons.A. We consider a multi-layer neural network combining complex-valued leaky neurons that integrate information over time (first layer) with prospective neurons (in the subsequent two layers). All neurons have prospective errors, following the prospective version of the $\delta$ dynamics in (\ref{['eqn:continuous-time-BP']}). The first layer's role is to store memories of past inputs, while the subsequent layers provide non-linear and instantaneous processing of these memories. B. We train the network on a delayed reaching task requiring production of context-dependent target trajectories. C. The network successfully solves the task when trained online with prospective real-time recurrent learning. When memory-specific parameters are frozen (prospective spatial backpropagation), the network can only adjust existing movements but cannot refine them to match the desired behavior. When the leaky neurons are replaced by prospective ones, the network loses its memory capacity and produces no movement as its best response. D. Learning memory-specific parameters becomes increasingly critical when the number of non-prospective leaky neurons is limited. As this number approaches infinity, the pressure decreases since the feedforward network in the last two layers has a wider range of movements to select from to solve the task.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2
• Theorem 1
• proof
• Theorem 2
• proof