Thermodynamic limit in learning period three

TL;DR

The paper investigates whether learning a period-three map can embed all periodic orbits in one-dimensional dynamics. It analyzes a one-layer random readout network in the thermodynamic limit, showing that almost all learned periods are unstable and that latent dynamics yield a spectrum of characteristic attractors, which can be externalized through a post-learning bifurcation controlled by feedback strength. When the interpolation is quadratic, a universal correspondence to the logistic map emerges, enabling universal externalization of all periods under appropriate conditions, with Sharkovsky-type ordering appearing in the bifurcation thresholds. The results establish LP3 as a versatile framework for embedding and controlling one-dimensional bifurcation structures in learning machines, with potential applications to physical neural networks and neuromorphic devices; they also identify symmetry and finite-size effects that shape the attractor landscape.

Abstract

A continuous one-dimensional map with period three includes all periods. This raises the following question: Can we obtain any types of periodic orbits solely by learning three data points? In this paper, we report the answer to be yes. Considering a random neural network in its thermodynamic limit, we first show that almost all learned periods are unstable, and each network has its own characteristic attractors (which can even be untrained ones). The latently acquired dynamics, which are unstable within the trained network, serve as a foundation for the diversity of characteristic attractors and may even lead to the emergence of attractors of all periods after learning. When the neural network interpolation is quadratic, a universal post-learning bifurcation scenario appears, which is consistent with a topological conjugacy between the trained network and the classical logistic map. In addition to universality, we explore specific properties of certain networks, including the singular behavior of the scale of weight at the infinite limit, the finite-size effects, and the symmetry in learning period three.

Figures (12)

• Figure 1: Trained maps $f_N^*$ for activation functions $\phi=\text{erf}$ (top left), $\sin$ (top right), $\cos$ (bottom left), and $\text{ReLU}$ (bottom right), with target period three ${\cal D}=\{-1,1,0\}$ and scale of weights $\sigma_w=\sigma_b=\sigma=1.0$. The blue-colored areas indicate the maximum--minimum regions of $f_N^*$ for 100 different realizations. The red circles and the red dotted lines show ${\cal D}$ and the thermodynamic limit $f_\infty^*$, respectively. The shade of blue corresponds to the number of nodes $N$, thus indicating that $f_N^*$ degenerates into $f_\infty^*$ as $N$ increases.
• Figure 2: Change in dynamical system $f_\infty^*$ for $\phi=\text{erf}$ with respect to $c$, with $a=-1$, $b=1$, and $\sigma=1.0$. The trajectory $\{x_n\}_{n=0}^{T}$ of $T=10^5$ steps is computed with different $x_0$ and $c$ in given intervals, excluding $c=a,b$. (a) Changes in the learned periods $\{x_p\}$ of period $p$, calculated by solving $(f_\infty^*)^{p}(x_p) = x_p$ (see Appendix \ref{['appendix:numerical']} for details). (b) Bifurcation diagram of the characteristic attractors calculated with $-10\leq x_0 \leq 10$. The dotted lines indicate ${\cal D}$; the diagonal lines correspond to varying $c$. (c) Change in the basin of attraction, where $f_\infty^*$ has multiple untrained attractors. (d) Change in the map $f_\infty^*$ in $-0.8 \leq c \leq 0.8$. The circles indicate ${\cal D}$, and the red circle indicates a c-independent point $(a,b)=(-1,1)$. As $c$ approaches $a$ or $b$, the folding of $f_\infty^*$ around ${\cal D}$ becomes larger, making the characteristic attractors wider.
• Figure 3: Change in dynamical systems $\sigma_\text{fb}f_\infty^*$ with respect to feedback strength $\sigma_\text{fb}$, with ${\cal D}=\{-1,1,-0.8\}$ and $\sigma=1$. (a) Bifurcation diagrams calculated with $-10\leq x_0 \leq 10$ and $T=10^5$. (b) Stability changes in the learned periods $\{x_p\}$ of period $p$, calculated by $\lambda_p \equiv \ln \left| \frac{d}{dx} (\sigma_\text{fb}f_\infty^*)^{p}(x_p) \right|$ using the solutions of $(\sigma_\text{fb}f_\infty^*)^{p}(x_p) = x_p$. The vertical dotted lines correspond to the network state in LP3 ($\sigma_\text{fb} = 1$). The horizontal dotted line indicates the boundary of the stability $\lambda_p = 0$; $\lambda_p < 0$ and $\lambda_p > 0$ mean (locally) stable and unstable, respectively. In the range $0 \leq \sigma_\text{fb} \leq 1$, the unstable periods at $\sigma_\text{fb} = 1$ will emerge as specific attracting periodic orbits, referred to as "externalized."
• Figure 4: Four possible types of mapping $\kappa(\sigma_\text{fb})$ (Eq. \ref{['eq:kappa']}) restricted to the interval $[0,1]$, depicted as colored regions in the $\beta\Delta$-plane. The insets provide examples for each type of $\kappa(\sigma_\text{fb})$ in the $\sigma_{\text{fb}}\kappa$-plane, corresponding to the colored points in the $\beta\Delta$-plane. The black solid lines in the insets indicate $\kappa([0,1])$ in the $\sigma_{\text{fb}}\kappa$-plane. Note that $\kappa(1) = \Delta + \frac{\beta}{2} \leq -\frac{7}{4}$, due to the presence of period three shi2007chaos, $\kappa(-\frac{\beta}{4\Delta})=-\frac{\beta^2}{16\Delta}$ (an extremum), and $\kappa(0) = 0$.
• Figure 5: Comparison of neural network interpolations (Eqs. \ref{['eq:fN']} and \ref{['eq:finf']}) with quadratic interpolation (Eq. \ref{['eq:quadratic']}) using $\sigma_b=1.0$ and ${\cal D}=\{-1,1,0\}$. (a) Relative errors $e$ (Eq. \ref{['eq:error']}) assessing the closeness of neural network interpolation to the quadratic interpolation. Colored areas indicate the maximum--minimum regions of the relative errors for 500 different realizations of $f_{N=100}^*$. Dotted lines correspond to their thermodynamic limit $N\rightarrow\infty$. (b) Change in the maps $f_\infty^*(x)$ with respect to $\sigma_w$, with $\phi=\text{erf}$ (left), $\sin$ (middle), and $\text{ReLU}$ (right). The red circles and the red dotted lines show ${\cal D}$ and the corresponding quadratic map, respectively. The insets are zoom-outs in the range $[-50, 150]$. (c) Externalization $\sigma_\text{fb}f_\infty^*(x)$ with $\sigma_w=10^{-1}$, calculated with $-10\leq x_0 \leq 10$, $T=10^4$, and $\varepsilon=10^{-6}$. With a small $\sigma_w$ and analytic $\phi$, the externalization shows the quantitative universality characterized by quadratic interpolation (see rightmost panel). However, it also generates attractors not explained by this universality, as shown in the zoomed-out bifurcation diagrams (insets) and in the region in which the attractors of the quadratic map vanishes due to a boundary crisis ($\sigma_\text{fb} \approx 1.05$) grebogi1983crises. The trained network with $\phi=\text{ReLU}$ does not converge to quadratic interpolation as $\sigma_w$ decreases, but it still exhibits externalization qualitatively similar to quadratic interpolation.

Theorems & Definitions (7)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Corollary 5
• Theorem 6
• Theorem 7