Automated Discovery of Operable Dynamics from Videos

TL;DR

This work tackles the challenge of automatically deriving compact, operable dynamical representations directly from video data. It introduces a two-stage auto-encoder pipeline that yields smooth neural state variables $\mathbf{V}_t$ in $\mathbb{R}^d$ and a differentiable neural state vector field $\hat{F}:\mathbb{R}^d\to\mathbb{R}^d$, enabling calculus-based analysis without domain priors. The framework identifies equilibria and their linearized dynamics, estimates natural frequencies, and distinguishes periodic, limit-cycle, and chaotic behaviors across four systems, while also enabling synthesis of novel, physically plausible dynamics by damping toward equilibria. Demonstrated across spring-mass, single and double pendulums, and cylinder wake, the method reveals rich non-equilibrium phenomena and robust long-term predictive capability, offering a data-driven bridge to classical scientific reasoning. By automatically discovering interpretable state representations from raw observations, this approach can accelerate automated scientific discovery and future applications in physics, chemistry, and biology.

Abstract

Dynamical systems form the foundation of scientific discovery, traditionally modeled with predefined state variables such as the angle and angular velocity, and differential equations such as the equation of motion for a single pendulum. We introduce a framework that automatically discovers a low-dimensional and operable representation of system dynamics, including a set of compact state variables that preserve the smoothness of the system dynamics and a differentiable vector field, directly from video without requiring prior domain-specific knowledge. The prominence and effectiveness of the proposed approach are demonstrated through both quantitative and qualitative analyses of a range of dynamical systems, including the identification of stable equilibria, the prediction of natural frequencies, and the detection of of chaotic and limit cycle behaviors. The results highlight the potential of our data-driven approach to advance automated scientific discovery.

Figures (18)

• Figure 1: The pipeline of our method to extract smooth neural state variables and neural vector field from videos. (a) Our framework automatically extracts compact and operable representations directly from observational data, provided as video frames. (b) Our minimally intrusive smoothness constraints enforce the neural state variables to exhibit smooth trajectories. (c) We trained an additional neural network $\hat{F}$ to represent the neural state vector field that describes the system dynamics of the discovered smooth neural state variables. Our discovered smooth neural state variables and neural state vector fields allow various scientific analyses, as exemplified in the following sub figures. (d) The identified stable equilibrium state and the decoded images of the single pendulum are marked within its neural state vector field; (e) Indicating chaotic behavior, two nearly identical initial states of the double pendulum exhibit diverging smooth neural state variable trajectories; (f) The developing oscillations in a smooth neural state variable trajectory indicate that the cylinder wake system's dynamics is attracted to a limit cycle, corresponding to the laminar periodic vortex shedding of the flow; (g) The damped neural state vector field of the spring mass system push integrated trajectories towards the stable equilibrium. More results on all systems can be seen in later sections.
• Figure 1: Physical parameters of the double pendulum system
• Figure 2: (a) Sample images of the four studied systems are shown with their respective intrinsic dimension: the spring mass ($d=2$), the single pendulum ($d=2$), the double pendulum ($d=4$), and the cylinder wake ($d=3$). (b) Visualizations of the respective discovered smooth neural state variables and neural state vector fields for the four studied systems demonstrate our framework's ability to extract smooth state trajectories that follow well defined continuous dynamics. (c) The baseline neural state variables and neural state vector fields trained on baseline neural state variables for the four systems demonstrate the highly disorganized state space when no regularization constraints are enforced. For the double pendulum and cylinder wake systems, which have intrinsic dimensions greater than two, we show the neural state vector field in the $V_1$ and $V_2$ dimensions.
• Figure 2: (a) Long-term prediction accuracy from rollout predictions (b) Long-term prediction accuracy through neural state vector field integration (c) Neural State Variable Auto-encoder Pixel Reconstruction Error (d) Neural State Vector Field Integration Single Step Pixel Reconstruction Error
• Figure 3: Near-equilibrium analysis. (a) The identified stable equilibrium states are decoded to two frames of images and marked with estimated physical values through computer vision techniques: position for the spring mass, angle for single and double pendulum, and normalized energy for the cylinder wake. (b) The stability of the equilibrium states is demonstrated for the spring mass, single pendulum and double pendulum systems. We sampled 6 initial states within a distance $\delta$ from the equilibrium ( $\delta=1\%$ for the spring mass and single pendulum, and $\delta=0.5\%$ for the double pendulum) and integrated their neural state vector fields within a region $\epsilon=1.5\%$ centered around the equilibrium state, where $\delta$ and $\epsilon$ are measured in proportion to the range of the neural state variables observed in the test data. The expected natural frequencies $\omega$ and the mean estimated natural frequencies $\hat{\omega}$ along with their respective standard errors are also shown below each system. (c) The mean absolute error of the predicted equilibrium states, along with their standard error bounds, are compared between the results from neural state vector fields trained on smooth and non-smooth neural state variables.