Train Stochastic Non Linear Coupled ODEs to Classify and Generate

TL;DR

The paper addresses the need for interpretable, dynamical alternatives to opaque neural networks by proposing a universal framework of coupled ODEs that can classify and generate data through attractor dynamics. It develops both deterministic and stochastic formulations, deriving that small-noise dynamics around planted attractors are Gaussian with covariances given by Lyapunov equations, and demonstrates how trainable coupling and noise improve robustness to perturbations. A complete generative pipeline is introduced, combining an encoder, dynamical latent evolution toward attractor-centered Gaussians, and a decoder, with losses that enforce attractor convergence, reconstruction, and centroid separation, enabling conditional generation and latent-space disentanglement. The approach yields a transparent, neuroscience-inspired alternative to deep networks, with practical demonstrations on MNIST and CelebA that show competitive generation quality and improved robustness via dynamic, sample-efficient training of the network’s interaction structure. Overall, the work contributes a principled, fully dynamical platform for joint classification and generation and outlines clear directions for scaling and extension to more complex data.

Abstract

A general class of dynamical systems which can be trained to operate in classification and generation modes are introduced. A procedure is proposed to plant asymptotic stationary attractors of the deterministic model. Optimizing the dynamical system amounts to shaping the architecture of inter-nodes connection to steer the evolution towards the assigned equilibrium, as a function of the class to which the item - supplied as an initial condition - belongs to. Under the stochastic perspective, point attractors are turned into probability distributions, made analytically accessible via the linear noise approximation. The addition of noise proves beneficial to oppose adversarial attacks, a property that gets engraved into the trained adjacency matrix and therefore also inherited by the deterministic counterpart of the optimized stochastic model. By providing samples from the target distribution as an input to a feedforward neural network (or even to a dynamical model of the same typology of the adopted for classification purposes), yields a fully generative scheme. Conditional generation is also possible by merging classification and generation modalities. Automatic disentanglement of isolated key features is finally proven.

Figures (19)

• Figure 1: A schematic layout to illustrate the deployment of model, via its iterative (\ref{['general_model']}) Euler (or Euler- Maruyama for the stochastic version) representation, on a residual feedforward neural network. Backpropagation through time is performed to propagate the gradients from the output - the late time stage of the dynamical evolution - to the input - the supplied initial condition.
• Figure 2: Example of corrupted MNIST images from the MNIST-C benchmark mu2019mnistc.
• Figure 3: A subset of trajectories, referring to specific entries of the state vector $x(t)$, are plotted against time, for the trained Continuous-Variable Firing Rate with $\epsilon=0$. As an initial condition we chose one image belonging to the MNIST test set. The input (here a $28 \times 28$ image of an 8) is unwrapped as a vector of size $784$ and normalized to its largest value. Colors are assigned as reflecting the final value that each pixel is expected to attain, based on the sculpted attractor. Classification is hence correctly performed.
• Figure 4: Same as in Figure \ref{['fig:evol_1']}, but for the stochastic version of the Continuous-Variable Firing Rate. In particular, we set $\epsilon=0.1$ and assume a matrix $G$ to coincide with the identity matrix. We are hence dealing with uncorrelated Gaussian noise, of amplitude $\epsilon$.
• Figure 5: Visual representation of the accuracy of different models ($\epsilon=0$, $\epsilon=0.05$ and $G= \mathbb{I}$, $\epsilon=0.1$ and $G= \mathbb{I}$, $\epsilon=1$ and $G$ fully trainable) trained on the uncorrupted MNIST model and tested against data gathered from MNIST-C. The reported accuracies are specifically referred to data perturbed with Gaussian noise with $\sigma \in \{0.5, 0.8\}$, shot noise, impulse noise, glass blur, motion blur, fog, Canny edges.