Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CHLU: The Causal Hamiltonian Learning Unit as a Symplectic Primitive for Deep Learning

Pratik Jawahar, Maurizio Pierini

TL;DR

It is shown that the CHLU is designed for infinite-horizon stability, as well as controllable noise filtering, and strictly conserves phase-space volume, as an attempt to solve the memory-stability trade-off.

Abstract

Current deep learning primitives dealing with temporal dynamics suffer from a fundamental dichotomy: they are either discrete and unstable (LSTMs) \citep{pascanu_difficulty_2013}, leading to exploding or vanishing gradients; or they are continuous and dissipative (Neural ODEs) \citep{dupont_augmented_2019}, which destroy information over time to ensure stability. We propose the \textbf{Causal Hamiltonian Learning Unit} (pronounced: \textit{clue}), a novel Physics-grounded computational learning primitive. By enforcing a Relativistic Hamiltonian structure and utilizing symplectic integration, a CHLU strictly conserves phase-space volume, as an attempt to solve the memory-stability trade-off. We show that the CHLU is designed for infinite-horizon stability, as well as controllable noise filtering. We then demonstrate a CHLU's generative ability using the MNIST dataset as a proof-of-principle.

CHLU: The Causal Hamiltonian Learning Unit as a Symplectic Primitive for Deep Learning

TL;DR

It is shown that the CHLU is designed for infinite-horizon stability, as well as controllable noise filtering, and strictly conserves phase-space volume, as an attempt to solve the memory-stability trade-off.

Abstract

Current deep learning primitives dealing with temporal dynamics suffer from a fundamental dichotomy: they are either discrete and unstable (LSTMs) \citep{pascanu_difficulty_2013}, leading to exploding or vanishing gradients; or they are continuous and dissipative (Neural ODEs) \citep{dupont_augmented_2019}, which destroy information over time to ensure stability. We propose the \textbf{Causal Hamiltonian Learning Unit} (pronounced: \textit{clue}), a novel Physics-grounded computational learning primitive. By enforcing a Relativistic Hamiltonian structure and utilizing symplectic integration, a CHLU strictly conserves phase-space volume, as an attempt to solve the memory-stability trade-off. We show that the CHLU is designed for infinite-horizon stability, as well as controllable noise filtering. We then demonstrate a CHLU's generative ability using the MNIST dataset as a proof-of-principle.
Paper Structure (26 sections, 8 equations, 8 figures, 1 algorithm)

This paper contains 26 sections, 8 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Relativistic Kinetic Governor:
  3. Action Propagation:
  4. Methodology: The Geometry of a CHLU
  5. The Separable Hamiltonian Engine
  6. Relativistic Kinetic Governor
  7. Symplectic Integration via Velocity Verlet
  8. Training Dynamics: Hamiltonian Contrastive Divergence
  9. Hamiltonian Annealing:
  10. Generative Sampling via Langevin Dynamics:
  11. Scope of Experiments and Results
  12. Experiment I: Long-Horizon Stability (Lemniscate Tracing)
  13. Experiment II: Kinetic Safety (The Perturbed Sine Wave)
  14. Experiment III: Thermodynamic Generation
  15. Conclusion: From Learning Dynamics to Enforcing Geometry
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Lemniscate tracing experiment. The transparent lines show inference cycles $0-48$ and the solid lines show the final $2$ cycles, for LSTM, NODE and CHLU (left to right)
  • Figure 2: The LSTM, NODE and CHLU (left to right), each predicts the state $q$ given the same initial perturbed states, from which we calculate the KE and overlay it on the expected KE.
  • Figure 3: MNIST digits generated from the centroid of the test set with added noise.
  • Figure 4: The learned lemniscate potential energy plotted as $2D$, $3D$ heatmaps (top left, right) as well as a force field (bottom). The output trajectory is overlayed on all $3$ for visual comparison.
  • Figure 5: Kinetic energies calculated from the output state $q$ (top 3 rows), compared to the kinetic energies calculated from output state $p$ (bottom 3 rows) for the same $3$ sine waves with the same inital perturbed conditions for all models (left to right: LSTM, NODE, CHLU).
  • ...and 3 more figures