Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

Nikolaos M. Matzakos

Abstract

We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs $\dot{h}=f_θ(h)$ with saturating activations ($\tanh$, sigmoid, etc.): if $q$ hidden layers of the MLP $f_θ$ satisfy $|σ'|\leδ$ on a region~$U$, the input Jacobian is attenuated as $\norm{Df_θ(x)}\le C(U)$ (for activations with $\sup_{x}|σ'(x)|\le 1$, e.g.\ $\tanh$ and sigmoid, this reduces to $C_Wδ^q$), forcing every Floquet (Lyapunov) exponen along any $T$-periodic orbit $γ\subset U$ into the interval $[-C(U),\;C(U)]$. This is a collapse of the Floquet spectrum: as saturation deepens ($δ\to 0$), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with $σ'>0$, a saturation-weighted spectral factorisation yields a refined bound $\widetilde{C}(U)\le C(U)$ whose improvement is amplified exponentially in~$T$ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of $\tanh$-NODEs on the Morris--Lecar neuron model.

Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

Abstract

We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs with saturating activations (, sigmoid, etc.): if hidden layers of the MLP satisfy on a region~, the input Jacobian is attenuated as (for activations with , e.g.\ and sigmoid, this reduces to ), forcing every Floquet (Lyapunov) exponen along any -periodic orbit into the interval . This is a collapse of the Floquet spectrum: as saturation deepens (), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with , a saturation-weighted spectral factorisation yields a refined bound whose improvement is amplified exponentially in~ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of -NODEs on the Morris--Lecar neuron model.

Paper Structure

This paper contains 34 sections, 23 theorems, 42 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background.
  3. The problem.
  4. Central thesis.
  5. Motivating observation.
  6. Main contributions.
  7. Related work.
  8. Classical dynamical systems theory.
  9. Expressivity of neural networks.
  10. Physics-informed and structure-preserving learning.
  11. Proof strategy.
  12. Organisation.
  13. Setting and Notation
  14. Notational convention.
  15. Standing assumption.
  16. ...and 19 more sections

Key Result

Proposition 2.1

For every $x\in\mathbb{R}^d$, where $D_k(x):=\mathrm{diag}\!\bigl(\sigma'(a_{k,1}(x)),\ldots,\sigma'(a_{k,d_k}(x))\bigr) \in\mathbb{R}^{d_k\times d_k}$. $\blacktriangleleft$$\blacktriangleleft$

Figures (5)

  • Figure 1: Jacobian attenuation (Theorem \ref{['thm:main']}).(a) Spectral norm $\lVert Df_\theta \rVert_2$ (solid blue) and upper bound $C_W(s)\cdot\delta(s)$ (dashed red) as functions of the pre-activation scale $s$. The shaded region is the gap between the bound and the true norm; the bound is satisfied for all $s$. (b) Decomposition of the bound: the weight factor $C_W(s)=s\,\lVert W_1 \rVert_2\,\lVert W_2 \rVert_2$ grows linearly in $s$, but the saturation factor $\delta(s)$ decays exponentially, so their product still vanishes.
  • Figure 2: Phase portraits: exact Stuart--Landau vs. Neural ODE at increasing saturation.Left: exact Stuart--Landau flow ($\ln\det M=-4\pi$, strongly attracting limit cycle). Panels 2--4: bias-shifted 256-unit MLP at $s=1,4,15$. Dashed circle: Stuart--Landau limit cycle ($r=1$). Annotations show $\delta$, $\ln\det M$, and $\det M$. At $s=1$ ($\delta\approx 0.53$, $\det M\approx 10^{-5}$) the MLP reproduces dissipative spirals; at $s=4$ ($\delta\approx 0.005$, $\det M\approx 0.92$) convergence weakens visibly; at $s=15$ ($\delta\approx 0$, $\det M=1$) trajectories are nearly rectilinear and the limit cycle no longer attracts.
  • Figure 3: Floquet--Liouville obstruction (Theorem \ref{['thm:liouville']}). The obstruction bound $d\,C(U;\,s)\,T$ on $|\ln\det M_\gamma|$ as a function of pre-activation scale $s$, for four bias-offset values $c\in\{0,\,1.5c_{\min},\,3c_{\min},\,6c_{\min}\}$ where $c_{\min}=\max_j\lVert W_{1j\cdot} \rVert_2$. Dotted line: the exact Stuart--Landau value $|\ln\det M_{\mathrm{SL}}|=4\pi$. The unbiased case ($c=0$, top curve) grows with $s$ because zero crossings keep $\delta=1$. Each positive offset eliminates zero crossings; with $c=6c_{\min}$ the bound falls below $10^{-2}$ at moderate $s$.
  • Figure 4: Individual Floquet multiplier bounds (Theorem \ref{['thm:individual']}). The grey band is the allowed window $[e^{-C(U)T},\,e^{C(U)T}]$; blue and red curves are $|\mu_1|,|\mu_2|$. As $s$ increases ($\delta\to 0$), the window collapses to $\{1\}$ and both multipliers converge to unity.
  • Figure 5: Refined bound verification (Illustration E). A $32$-unit $\tanh$ MLP is trained at $s{=}1$; weights are frozen and $s$ is swept from $1$ to $30$. (a) Actual Jacobian norm, refined bound, and original bound on a log scale. (b) Improvement ratio (original / refined) grows monotonically to $9\times$ at $s=30$.

Theorems & Definitions (46)

  • Proposition 2.1: Jacobian factorization; standard, see e.g. Goodfellow2016
  • Definition 2.2
  • Remark 2.3: Choice of $U$
  • Remark 2.4: When does saturation occur?
  • Lemma 3.1: Grönwall Hartman1964Chicone2006
  • Lemma 3.2: Trajectory sensitivity; standard, see e.g. Hartman1964
  • Theorem 3.3: Saturation-induced Jacobian attenuation
  • proof
  • Remark 3.4
  • Remark 3.5: Tightness
  • ...and 36 more