Inferring Kernel $ε$-Machines: Discovering Structure in Complex Systems

TL;DR

This work expands the empirical kernel causal state algorithm to explicitly introduce the causal diffusion components it produces, which encode the kernel causal state estimates as a set of coordinates in a reduced dimension space.

Abstract

Previously, we showed that computational mechanic's causal states -- predictively-equivalent trajectory classes for a stochastic dynamical system -- can be cast into a reproducing kernel Hilbert space. The result is a widely-applicable method that infers causal structure directly from very different kinds of observations and systems. Here, we expand this method to explicitly introduce the causal diffusion components it produces. These encode the kernel causal-state estimates as a set of coordinates in a reduced dimension space. We show how each component extracts predictive features from data and demonstrate their application on four examples: first, a simple pendulum -- an exactly solvable system; second, a molecular-dynamic trajectory of $n$-butane -- a high-dimensional system with a well-studied energy landscape; third, the monthly sunspot sequence -- the longest-running available time series of direct observations; and fourth, multi-year observations of an active crop field -- a set of heterogeneous observations of the same ecosystem taken for over a decade. In this way, we demonstrate that the empirical kernel causal-states algorithm robustly discovers predictive structures for systems with widely varying dimensionality and stochasticity.

Figures (11)

• Figure 1: Probability distributions over pasts $X$ and conditional distributions over futures $Y$ can be represented as points in the reproducing kernel Hilbert spaces $\mathcal{H} ^{ X }$ and $\mathcal{H} ^{ Y }$, respectively.
• Figure 2: The empirical kernel causal state algorithm. Step 1: Build a library of past sequences $x _i$ and future sequences $y _i$ from observed data. Step 2: Construct an appropriate kernel for pasts $k ^{ X }$ and a kernel for futures $k ^{ Y }$. Embed pasts (circles) and futures (squares) into their respective Hilbert spaces, $\mathcal{H} ^{ X }$ (blue) and $\mathcal{H} ^{ Y }$ (red). Step 3: Calculate the empirical kernel causal state embedding coefficients $\alpha _i$ for a new past $x$ by comparing $x$ to each embedded past $x _i$. More similar pasts will have larger embedding coefficients. Step 4: Construct the distribution over futures for $x$ using the calculated $\alpha _i$ to weight the sum over the associated futures $y _i$.
• Figure 3: Relation between the causal diffusion components $\psi$---the eigenvectors of the diffusion matrix $M^{ \mathcal{S} _{ k } }$---and the coordinates that encode the empirical kernel causal states.
• Figure 4: Left: The simple pendulum with angular velocity $\dot{ \theta }$ and angle from the vertical $\theta$. Middle: Pendulum trajectories in the angle and angular velocity $\left( \theta , \dot{ \theta } \right)$ phase space at different energy levels. The dotted line is the separatrix, for which the total energy $E = E _{\text{sep}}$. The ellipsoid trajectories in the middle represent trajectories with $E < E _{\text{sep}}$ and the trajectories above and below the separatrix have $E > E _{\text{sep}}$. The red and blue trajectories were used as input to the empirical kernel causal state algorithm. Right: Each resultant color coded set of empirical kernel causal states plotted using their first two diffusion coordinates. Eigenvalues and spectral gaps are given in \ref{['fig:pendulum_eigenvalues']}.
• Figure 5: Eigenvalues and spectral gaps for the simple pendulum empirical kernel causal states. The two conservative cases match those in \ref{['fig:undampedPendulum']} and the dissipative case \ref{['fig:dampedPendulum']}.