Constraint Breeds Generalization: Temporal Dynamics as an Inductive Bias

TL;DR

This work addresses how temporal structure and energy constraints can improve generalization by introducing a temporal inductive bias. It advances a dual-path framework—external dynamical encoding via a Duffing oscillator and internal architectural dissipation via leaky neurons—aimed at achieving stable, invariant representations through controlled phase-space contraction. Across classification, reconstruction, and zero-shot reinforcement learning, the authors identify a transition regime that maximizes generalization, fosters structured feature emergence, and aligns with Slow Feature Analysis through a low-frequency, high-entropy spectrum. A PAC-Bayesian perspective formalizes why these temporal constraints act as a favorable prior, suggesting that robust AI arises not solely from scaling but from mastering the temporal dynamics that govern learning and generalization.

Abstract

Conventional deep learning prioritizes unconstrained optimization, yet biological systems operate under strict metabolic constraints. We propose that these physical constraints shape dynamics to function not as limitations, but as a temporal inductive bias that breeds generalization. Through a phase-space analysis of signal propagation, we reveal a fundamental asymmetry: expansive dynamics amplify noise, whereas proper dissipative dynamics compress phase space that aligns with the network's spectral bias, compelling the abstraction of invariant features. This condition can be imposed externally via input encoding, or intrinsically through the network's own temporal dynamics. Both pathways require architectures capable of temporal integration and proper constraints to decode induced invariants, whereas static architectures fail to capitalize on temporal structure. Through comprehensive evaluations across supervised classification, unsupervised reconstruction, and zero-shot reinforcement learning, we demonstrate that a critical "transition" regime maximizes generalization capability. These findings establish dynamical constraints as a distinct class of inductive bias, suggesting that robust AI development requires not only scaling and removing limitations, but computationally mastering the temporal characteristics that naturally promote generalization.

Figures (7)

• Figure 1: Cross-encoding generalization with hierarchical variance reduction.(a-e) 3D generalization landscapes (mean accuracy with deviation, $n=10$). (c) SNNs exhibit a robust generalization in the transition regime ($\delta \approx 0-2.0$) across the spectrum. This contrasts with the diagonal-only ridges seen in expansive regimes ($\delta < 0$) and in baseline architectures (Last-T MLP, LSTM, RNN; b, d, e), and Avg-Pool MLP (a) shows partial, unstable generalization (see Appendix Tables \ref{['tab:cross_enc_mlp_avg']}--\ref{['tab:cross_enc_rnn']}). (f) Out-of-Distribution accuracy strongly anti-correlates with Layer 1 neural variability (CV; $r=-0.962$). Error bars: $\pm 1$ SEM. (g) Hierarchical variance reduction (L1 $\to$ L3) emerges in SNN across regimes.
• Figure 2: Spontaneous emergence of structured receptive fields. Visualization of learned features (receptive fields) at the network bottleneck. Red and blue pixels represent positive (excitatory) and negative (inhibitory) weights, respectively. Note that clear spatial antagonism emerges uniquely under Transition dynamics, whereas other regimes yield unstructured noise.
• Figure 3: Dynamical Universality Validation. We verify the constraint principle across two alternative dynamical systems to disentangle the effect of dissipation from specific equation geometries. (a) Lorenz System: Structure emerges in the stable regime (low $\rho$) but collapses into noise as the system enters the classical chaotic regime ($\rho=28$). (b) Thomas Attractor: At low damping ($b < 0.2$), hyper-chaotic dynamics prevent learning. As dissipative constraint increases ($b \to 1.0$), forcing the system into a transition regime, structured features spontaneously emerge. Consistent results across polynomial (Lorenz) and sinusoidal (Thomas) non-linearities confirm that phase space contraction is the universal driver of structural generalization.
• Figure 4: Intrinsic Dynamics and the Emergence of Structure. We analyze the impact of dynamical constraints ($\delta$) on feature organization across three sparsity regimes: (a) Standard constraint ($\lambda=1.0$), (b) Weak constraint ($\lambda=0.1$), and (c) No constraint ($\lambda=0.0$). While a "structural ridge" ($\delta \in [1, 3]$) persists across all regimes, the peak structural organization increases as the explicit sparsity constraint decreases (from $\sim 0.47$ at $\lambda=1.0$ to $\sim 0.61$ at $\lambda=0.0$). This trend is visually corroborated by the receptive field visualizations (bottom panels), where the diversity and clarity of structured filters visibly expand as the external constraint is relaxed. This demonstrates that the transition dynamics alone serve as a novel intrinsic inductive bias compared to external regularization.
• Figure 5: Mechanism of Spectral Alignment and Invariance.(a) Scale-space heatmaps across varying physical timescales ($T_{max}$) demonstrate the robustness of this mechanism. The dark vertical band in the transition region indicates that the "Low-Frequency, High-Structure" valley is scale-invariant, persisting stably across different observation windows. This confirms that the generalization capability arises from locking onto intrinsic invariants rather than transient artifacts. (b) Spectral analysis reveals a unique signature in the Transition regime ($\delta \approx2$): it minimizes the Frequency Centroid (bottom) to align with the network's spectral bias, while simultaneously maintaining high Spectral Entropy (top) to preserve structural complexity.