Deep Learning-based Approaches for State Space Models: A Selective Review
Jiahe Lin, George Michailidis
TL;DR
This selective review surveys deep learning-based state-space models (SSMs) for dynamical systems, covering both discrete-time and continuous-time formulations such as latent neural ODEs and latent neural SDEs. It distinguishes partially DL approaches that augment classical SSM learning from fully DL pipelines that use variational autoencoders to learn latent dynamics end-to-end, and discusses the role of encoder/decoder architectures in capturing latent trajectories. The article highlights powerful DL modules for long-range sequence modeling, including linear state-space layers (LSSLs) and structured SSM variants like S4, S5, and Mamba, augmented by HiPPO memorization and efficient GPU implementations. It also discusses practical applications to mixed-frequency and irregularly spaced time series, demonstrating the advantages of SSMs in handling complex temporal dependencies and missing data, and outlines future directions for scalable, expressive DL-based dynamical modeling.
Abstract
State-space models (SSMs) offer a powerful framework for dynamical system analysis, wherein the temporal dynamics of the system are assumed to be captured through the evolution of the latent states, which govern the values of the observations. This paper provides a selective review of recent advancements in deep neural network-based approaches for SSMs, and presents a unified perspective for discrete time deep state space models and continuous time ones such as latent neural Ordinary Differential and Stochastic Differential Equations. It starts with an overview of the classical maximum likelihood based approach for learning SSMs, reviews variational autoencoder as a general learning pipeline for neural network-based approaches in the presence of latent variables, and discusses in detail representative deep learning models that fall under the SSM framework. Very recent developments, where SSMs are used as standalone architectural modules for improving efficiency in sequence modeling, are also examined. Finally, examples involving mixed frequency and irregularly-spaced time series data are presented to demonstrate the advantage of SSMs in these settings.