Pulse-Driven Neural Architecture: Learnable Oscillatory Dynamics for Robust Continuous-Time Sequence Processing

TL;DR

PDNA (Pulse-Driven Neural Architecture) is introduced, a method for augmenting continuous-time recurrent networks with learnable oscillatory dynamics that maintain internal state evolution independently of external input that significantly improves robustness to input interruptions.

Abstract

We introduce PDNA (Pulse-Driven Neural Architecture), a method for augmenting continuous-time recurrent networks with learnable oscillatory dynamics that maintain internal state evolution independently of external input. Built on Closed-form Continuous-time (CfC) networks, PDNA adds two components: (1) a pulse module that generates structured oscillations $A \cdot \sin(ωt + \varphi(h))$ with learnable frequencies and state-dependent phase, and (2) a self-attend module that applies recurrent self-attention to the hidden state. Through a controlled ablation study on sequential MNIST (sMNIST) with five random seeds, we evaluate gap robustness -- the ability to maintain performance when portions of the input sequence are removed at test time. Our key finding is that structured oscillatory dynamics significantly improve robustness to input interruptions: the self-attend variant achieves a statistically significant 2.78 percentage point multi-gap advantage over baseline ($p = 0.041$), while the pulse variant shows a 4.62 pp advantage with large effect size (Cohen's $d = 0.87$). A noise control (random perturbation of equal magnitude) provides no benefit, confirming that the advantage is structural rather than merely dynamic. These results provide evidence that continuous-time models can benefit from biologically-inspired internal oscillatory mechanisms for temporal robustness.

Figures (7)

• Figure 1: The PDNA architecture. Input sequences are processed by the CfC backbone, producing hidden states $h_\text{cfc}$. The pulse module (left) adds structured oscillations $\alpha \cdot A \cdot \sin(\omega t + \varphi(h))$ with learnable per-dimension frequency $\omega$, amplitude $A$, state-dependent phase $\varphi(h)$, and scalar gate $\alpha$. The self-attend module (right) adds a gated self-projection $\beta \cdot W_\text{self} \cdot \sigma(h)$. Both are additive residuals with learned gates initialized at 0.01. The last timestep's hidden state is passed to a linear classifier.
• Figure 2: Accuracy under increasing gap severity on sMNIST (5 seeds, mean $\pm$ std bands). Pulse-augmented variants (C, E) degrade more gracefully than baseline, particularly on the multi-gap condition where scattered interruptions test recovery ability.
• Figure 3: Learned pulse parameters. (a) Oscillation frequencies span two orders of magnitude (median $1.02$, IQR $[0.31, 3.17]$), with a right-skewed distribution indicating most dimensions learn low frequencies. (b) The pulse strength $\alpha$ grows $\sim$66$\times$ from initialization.
• Figure 4: Effective pulse magnitude $\alpha \cdot \|A\|_2$ over training epochs (mean $\pm$ std, 5 seeds). Left: the product $\alpha \cdot \|A\|_2$ grows monotonically from 0.011 to $\sim$4.8 ($\sim$420$\times$). Center: $\alpha$ alone (0.01 $\to$ 0.67). Right: $\|A\|_2$ alone (1.14 $\to$ 7.17). Both components increase, ruling out the hypothesis that $\alpha$ growth is offset by amplitude shrinkage.
• Figure 5: Left: Training convergence on sMNIST (mean $\pm$ std, 5 seeds). All variants converge to similar final accuracy, confirming the pulse does not interfere with standard learning. Right: Multi-gap robustness comparison. Pulse-augmented variants (C, D, E) significantly outperform baseline and noise control. $p$-values from paired $t$-tests.