Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LyTimeT: Towards Robust and Interpretable State-Variable Discovery

Kuai Yu, Crystal Su, Xiang Liu, Judah Goldfeder, Mingyuan Shao, Hod Lipson

TL;DR

LyTimeT addresses the problem of discovering true dynamical variables from high-dimensional video by combining a TimeSformer-based, distraction-robust encoder with a Lyapunov stability regularizer. In Phase 1, it learns robust latent dynamics and multi-step predictions; in Phase 2, it extracts interpretable variables via linear probing and enforces contraction to stabilize roll-outs. Across five synthetic and four real-world dynamical systems, LyTimeT achieves intrinsic-dimension estimates close to ground truth, the lowest long-horizon error, and robust performance under nuisance perturbations, outperforming NSV and TIDE. The Lite variant offers substantial efficiency for real-time deployment, highlighting the practical impact for scientific discovery and data-driven modeling of complex dynamical systems.

Abstract

Extracting the true dynamical variables of a system from high-dimensional video is challenging due to distracting visual factors such as background motion, occlusions, and texture changes. We propose LyTimeT, a two-phase framework for interpretable variable extraction that learns robust and stable latent representations of dynamical systems. In Phase 1, LyTimeT employs a spatio-temporal TimeSformer-based autoencoder that uses global attention to focus on dynamically relevant regions while suppressing nuisance variation, enabling distraction-robust latent state learning and accurate long-horizon video prediction. In Phase 2, we probe the learned latent space, select the most physically meaningful dimensions using linear correlation analysis, and refine the transition dynamics with a Lyapunov-based stability regularizer to enforce contraction and reduce error accumulation during roll-outs. Experiments on five synthetic benchmarks and four real-world dynamical systems, including chaotic phenomena, show that LyTimeT achieves mutual information and intrinsic dimension estimates closest to ground truth, remains invariant under background perturbations, and delivers the lowest analytical mean squared error among CNN-based (TIDE) and transformer-only baselines. Our results demonstrate that combining spatio-temporal attention with stability constraints yields predictive models that are not only accurate but also physically interpretable.

LyTimeT: Towards Robust and Interpretable State-Variable Discovery

TL;DR

LyTimeT addresses the problem of discovering true dynamical variables from high-dimensional video by combining a TimeSformer-based, distraction-robust encoder with a Lyapunov stability regularizer. In Phase 1, it learns robust latent dynamics and multi-step predictions; in Phase 2, it extracts interpretable variables via linear probing and enforces contraction to stabilize roll-outs. Across five synthetic and four real-world dynamical systems, LyTimeT achieves intrinsic-dimension estimates close to ground truth, the lowest long-horizon error, and robust performance under nuisance perturbations, outperforming NSV and TIDE. The Lite variant offers substantial efficiency for real-time deployment, highlighting the practical impact for scientific discovery and data-driven modeling of complex dynamical systems.

Abstract

Extracting the true dynamical variables of a system from high-dimensional video is challenging due to distracting visual factors such as background motion, occlusions, and texture changes. We propose LyTimeT, a two-phase framework for interpretable variable extraction that learns robust and stable latent representations of dynamical systems. In Phase 1, LyTimeT employs a spatio-temporal TimeSformer-based autoencoder that uses global attention to focus on dynamically relevant regions while suppressing nuisance variation, enabling distraction-robust latent state learning and accurate long-horizon video prediction. In Phase 2, we probe the learned latent space, select the most physically meaningful dimensions using linear correlation analysis, and refine the transition dynamics with a Lyapunov-based stability regularizer to enforce contraction and reduce error accumulation during roll-outs. Experiments on five synthetic benchmarks and four real-world dynamical systems, including chaotic phenomena, show that LyTimeT achieves mutual information and intrinsic dimension estimates closest to ground truth, remains invariant under background perturbations, and delivers the lowest analytical mean squared error among CNN-based (TIDE) and transformer-only baselines. Our results demonstrate that combining spatio-temporal attention with stability constraints yields predictive models that are not only accurate but also physically interpretable.
Paper Structure (13 sections, 11 equations, 1 figure, 4 tables)

This paper contains 13 sections, 11 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Phase 1: Video Prediction with LyTimeT
  4. Phase 2: Variable Extraction and Lyapunov Stability
  5. Evaluation
  6. Experimental Setup
  7. Datasets
  8. Evaluation Metrics
  9. Comparation Experiments
  10. Ablation Study
  11. Encoder Variants
  12. Variable Extraction and Lyapunov
  13. Conclusion

Figures (1)

  • Figure 1: Overview of LyTimeT. Phase 1 (left) performs distraction-robust video prediction using a TimeSformer or Light version encoder that factorizes temporal and spatial attention, followed by mean pooling into a compact latent state $z_t$, a lightweight decoder for frame reconstruction, and a latent transition model $f_\theta$ for $K$-step roll-outs. Phase 2 (right) extracts interpretable variables $\tilde{z}_t$ from $z_t$ by linear probing and ranking, validates disentanglement across nuisance settings, and refines dynamics with a Lyapunov loss that enforces contractive trajectories for stable long-horizon prediction.