Primate-like perceptual decision making emerges through deep recurrent reinforcement learning

TL;DR

This work investigates why primate-like decision mechanisms emerge by training end-to-end recurrent reinforcement learning agents to perform a noisy perceptual discrimination task. Using a random-dot motion paradigm, the authors show that agents develop speed-accuracy tradeoffs, internal dynamics resembling evidence accumulation, and the ability to change decisions in response to new information, across both saccadic and continuous-arm actions. Decoding analyses reveal momentary and accumulated evidence signals in CNN and LSTM units, and changes-of-mind in neural-like trajectories and movement paths mirror primate findings, with ablations demonstrating recurrence and environmental noise as critical pressures. The study links artificial agent behavior to primate neurophysiology, supporting theories that decision-making mechanisms arise to maximize reward under temporal uncertainty, and offers insights for robust, flexible AI in dynamic environments.

Abstract

Progress has led to a detailed understanding of the neural mechanisms that underlie decision making in primates. However, less is known about why such mechanisms are present in the first place. Theory suggests that primate decision making mechanisms, and their resultant behavioral abilities, emerged to maximize reward in the face of noisy, temporally evolving information. To test this theory, we trained an end-to-end deep recurrent neural network using reinforcement learning on a noisy perceptual discrimination task. Networks learned several key abilities of primate-like decision making including trading off speed for accuracy, and flexibly changing their mind in the face of new information. Internal dynamics of these networks suggest that these abilities were supported by similar decision mechanisms as those observed in primate neurophysiological studies. These results provide experimental support for key pressures that gave rise to the primate ability to make flexible decisions.

Figures (11)

• Figure 1: Task and agent network architecture. a) Random dot motion discrimination stimuli with varying levels of noise. Dots at $t_{0}$ (white) either move coherently in one direction, or are replaced randomly at $t_{1}$ (red for illustration only) with an independent probability determined by coherence. All examples are from trials where the correct response is "right”. b) Saccadic random dot motion task structure. Agents are rewarded with +1 for responding correctly during the stimulus period, -0.1 for responding before or after the stimulus period, and 0 otherwise. c) Actor-critic agent architecture. Pixel and vector input are passed left to right through layers of a deep neural network consisting of convolutional and sum pool operations (CNN), a recurrent layer (LSTM), and fully-connected (FC) layers. Critic output is a single linear unit. Actor output depends on the action space of the task. d) Agents responding with saccades have access to discrete "left”, "wait”, and "right” actions. Agents responding with continuous arm movements control shoulder and elbow joint forces, parameterized by alpha ($\alpha$) and beta ($\beta$) parameters for independent beta distributions (see \ref{['methods']}). After training, network parameters were frozen and performance was analyzed.
• Figure 2: Property 1: Agents learn primate-like decision making behaviors. Agents were trained on the random dot motion task to respond either with discrete actions (left/right saccades), or continuous actions (controlling the shoulder and elbow forces of a simulated arm). Saccade agents were trained at either 60 Hz or 180 Hz, as the 180 Hz agents allow for higher temporal resolution to investigate dynamics (see \ref{['methods']}). Results did not differ between 60 and 180 Hz agents. a) Evidence accumulation model fit to the behavior of a single non-human primate's accuracy and b) response times. Primates completed the random dot motion task and were rewarded with an emphasis on either speed or accuracy hanks2014neural. c) Logistic functions fit to responses grouped by agent discount rate ($\gamma$; N = 15 per $\gamma$). Agents trained with higher discount rates (light gray) were more accurate than those trained with lower discount rates (dark gray). Models are 60 Hz saccade agents. d) Average response time by agent discount rate. Agents trained with higher discount rates (light gray) were slower to respond than those trained with lower discount rates (dark gray). e) Periodic evaluation of agents during training (colored traces), and final evaluation after training (dots). Shaded areas (traces) and vertical bars (dots) represent standard errors across 10 random seeds. Horizontal line at 50% accuracy represents chance performance on the random dot motion task for an agent that always responds within the stimulus period. Horizontal line at 80.3% represents maximum accuracy achievable from considering only a single time step of dot motion by a hand-constructed evidence accumulation model (see \ref{['methods']}). f) Slopes were extracted from logistic functions fit to each agent’s final evaluation accuracy. g) Indifference points were extracted from logistic functions fit to each agent's final evaluation accuracy. h) Mean response time by discount rate condition. Panels a and b are adapted from non-human primate data in hanks2014neural.
• Figure 3: Property 2: Agents learn primate-like internal decision making dynamics. Internal dynamics of a representative agent in the 180 Hz saccade task. Solid lines indicate rightward motion trials, and dashed lines indicate leftward motion trials. Data are from correct trials only. For dynamics on the left, individual trials making up the plotted averages are considered up until the first of: response time or median response time. a) Average response of direction selective MT neurons recorded in a primate during the random dot motion task gold2007neural. b) An example CNN unit with average activity proportional or inversely proportional to momentary motion coherence and direction, similar to primate area MT. c) The average activity of this CNN unit is roughly linearly related to motion coherence. d) Average response of LIP neurons recorded in a primate during the random dot motion task gold2007neural. On the left, neural activity is time-locked to dot motion onset. On the right, activity is time-locked to primate saccadic response. e) An example LSTM unit with activity proportional or inversely proportional to accumulated momentary motion coherence and direction, similar to primate area LIP. f) The average buildup slope, from dot motion onset to the first step of full motion, of this LSTM unit is roughly linearly related to coherence.
• Figure 4: Property 3: Decoding of neural changes of mind. a) Logistic decoder decision variable (DV) by coherence condition from a representative agent trained in the 180 Hz saccadic task. Shaded regions denote standard errors. Traces plotted until median response time. Solid traces indicate rightward motion trials, and dashed traces indicate leftward motion trials. b) Logistic decoder DV by coherence condition pooled across two non-human primates. Traces are plotted for the full time, as the primate needed to wait for a decision cue before response peixoto2021decoding. c) Example of single-trial DV traces. Line colors correspond to the coherence conditions of each single trial. Shaded region denotes change of mind threshold (see \ref{['methods']}). d) Example of two single-trial DV traces decoded from a non-human primate that were classified as change of mind trials. e) Decoded changes of mind. Agents displayed more corrective (green) than erroneous (red) changes of mind. f) Decoded changes of mind frequency from a non-human primate. g) Decoded changes of mind by coherence. Corrective changes (green) are those where the model DV at some point predicts an incorrect choice before an ultimately correct choice is made. Erroneous changes (red) are those where the model DV predicts a correct choice before an ultimately incorrect choice is made. h) Frequency of decoded changes of mind by coherence in a non-human primate. Bottom panels (b, d, f, and h) are adapted from non-human primate data in peixoto2021decoding.
• Figure 5: Property 4: Behavioral changes of mind. Comparison between human reaching performance (left column; N = 13) and trained deep reinforcement learning agents controlling a 2DOF arm (right column; N = 10). Individual representative human and trained agent highlighted (black line, coloured dots), and all other individuals (gray lines). a, b) Accuracy. Logistic function fits; humans: $R^{2} = 0.957 \pm 0.019$; artificial trained agents: $R^{2} = 0.998 \pm 0.001$. c, d) Response time. Note differences in scaling between humans and artificial agents. e, f) Human post-decision confidence ratings, and network critic output at response time. g, h) Mean changes of mind across all subjects. Vertical lines indicate standard errors. i, j) Movement trajectories from an individual representative human and artificial agent (2DOF arm shown), with selected changes of mind highlighted.