Phase codes emerge in recurrent neural networks optimized for modular arithmetic

TL;DR

The paper investigates whether phase codes can emerge in RNNs trained for modular arithmetic without explicit oscillatory biases. Using continuous-time RNNs trained on a modulo-3 arithmetic (M3A) task, they observe phase-code solutions in a subset of networks and map a broad solution space with Dynamical Similarity Analysis. They identify multiple mechanistic strategies beyond phase coding—hexagon fixed-point attractors, lattice attractors, star-like fast oscillations, and line attractors—and report that only a minority exhibit phase codes. The findings show phase coding can be a natural, not universal, outcome of task optimization and reveal a rich landscape of alternative dynamical solutions in simple tasks.

Abstract

Recurrent neural networks (RNNs) can implement complex computations by leveraging a range of dynamics, such as oscillations, attractors, and transient trajectories. A growing body of work has highlighted the emergence of phase codes, a type of oscillatory activity where information is encoded in the relative phase of network activity, in RNNs trained for working memory tasks. However, these studies rely on architectural constraints or regularization schemes that explicitly promote oscillatory solutions. Here, we investigate whether phase coding can emerge purely from task optimization by training continuous-time RNNs to perform a simple modular arithmetic task without oscillatory-promoting biases. We find that in the absence of such biases, RNNs can learn phase code solutions. Surprisingly, we also uncover a rich diversity of alternative solutions that solve our modular arithmetic task via qualitatively distinct dynamics and dynamical mechanisms. We map the solution space for our task and show that the phase code solution occupies a distinct region. These results suggest that phase coding can be a natural but not inevitable outcome of training RNNs on modular arithmetic, and highlight the diversity of solutions RNNs can learn to solve simple tasks.

Figures (4)

• Figure 1: Schematic of the modulo-3 arithmetic (M3A) task. The network receives inputs through three distinct channels, each representing one of the integers $0$, $1$, or $2$. On each trial, three integers are presented sequentially as brief pulses. The network’s task is to output $+1$ if the sum of the integers is congruent to $0$ modulo $3$, and $-1$ otherwise.
• Figure 2: Phase code solution in an RNN trained on the M3A task.(a) Network output for a congruent trial (top row) and the corresponding phase difference relative to null activity (bottom row). The network outputs $+1$ at the end of the trial, correctly indicating a congruent input. The phase difference responds to integer pulses, settling near zero by trial end. (b) Same as panel (a), but for an incongruent trial. (c) Network activity projected onto the first two principal components for the example trials in panels (a) and (b). Trajectories evolve along a limit cycle, with integer pulses inducing phase shifts (advances or delays). Null activity is shown as a dashed gray line. (d) Endpoints ($t=T$) of testing trials in PCA space cluster along the null activity in three distinct regions. Each cluster corresponds to a final sum congruent to $0$, $1$, or $2$ modulo 3. (e) Phase response curves (PRCs) for integer pulses $0$ (green), $1$ (red), and $2$ (purple). Each PRC shows a consistent shift in phase regardless of when the input is presented. Error bars represent standard deviation. (f) Summed mean PRC values for each integer sequence predict summed shifts in network phase. These summed phase values recapitulate the clustering observed in panel (d).
• Figure 3: Heterogeneity in the solution space of RNNs trained on the M3A task. An architecture hyperparameter search reveals that phase coding is just one of many viable strategies for solving the M3A task. (a) MDS projection of Dynamical Similarity Analysis (DSA) embeddings for 91 trained RNNs. Each point represents one trained model. The gold diamond marks the phase code solution from Fig. \ref{['fig:phase_code']}, and colored diamonds highlight models explored in panels (b–e). (b) PCA endpoint plot for the hexagon solution (green diamond). Trial endpoints converge to fixed-point attractors arranged in a hexagon pattern. Integer pulses transition the network between these attractors in a clock-like fashion. (c) Same as (b), but for the lattice solution (pink diamond). Attractors are arranged in a lattice-like configuration, and integer pulses drive vector-like transitions across the lattice. (d) Same as (b), but for the star solution (red diamond). The network exhibits fast oscillations that produce a star-like null activity. Unlike the phase code solution (Fig. \ref{['fig:phase_code']}d), endpoints form nine clusters instead of three. (e) Same as (b), but for the line solution (purple diamond). Attractors lie along a one-dimensional axis and each integer pulse shifts network activity along this axis.
• Figure 4: Training dynamics of the phase code solution.Left: Training loss (mean squared error) over 500 epochs. Right: Percentage of correct classifications during training. The network converges rapidly to near-zero error and near-perfect accuracy after approximately 150 epochs.