Simultaneous Latent State Estimation and Latent Linear Dynamics Discovery from Image Observations

TL;DR

This work addresses estimating latent state trajectories from high-dimensional image observations under near-linear latent dynamics. It surveys traditional filters and neural approaches, then introduces Normalizing Flows based Particle Filter (NFPF), which uses a conditional NF to model the observation likelihood $p(\mathbf{y}|\mathbf{x})$ and learns time-varying linear dynamics via neural networks. The framework jointly optimizes NF parameters, latent-state projections, and dynamic matrices, enabling full posterior filtering rather than only samples. While offering a principled density-based approach, the method faces computational and data limitations, highlighting a trade-off between SINDy-style latent-dynamics discovery and NF-based posterior estimation for image-based state estimation.

Abstract

The problem of state estimation has a long history with many successful algorithms that allow analytical derivation or approximation of posterior filtering distribution given the noisy observations. This report tries to conclude previous works to resolve the problem of latent state estimation given image-based observations and also suggests a new solution to this problem.

Figures (5)

• Figure 1: Autoencoder architecture: here $\mathbf{x}(t)$ stands for high-dimensional observation, $\mathbf{z}(t)$ stands for latent state and $\mathbf{\hat{x}}(t)$ is reconstructed observation by the decoder $\psi$.
• Figure 2: Proposed model architecture: $\mathbf{y}_t$ denotes image observation, $\hat{\mathbf{y}}_t$ denotes flow base distributed variables (Gaussian), $\mathbf{x}_t$ denotes latent state, $\mathbf{A}_t$ denotes dynamics matrix, $\mathbf{u}_t$ denotes control signal, $\mathbf{B}_t$ denotes control matrix, $\mu_0$ and $\Sigma_0$ are mean and covariance of the first latent state. Linear dynamics matrices are obtained using dynamics network $f_{\psi}$, $g_{\theta}$ denotes NF transformation, while $\mu_{\phi}$ stands for mean parametrization for $\hat{\mathbf{y}}$
• Figure 3: RGB observation of CartPole used in the experiments.
• Figure 4: Particle filter with 2 particles NF observation likelihood against true state evolution given the same control inputs for 4 dimensions.
• Figure 5: PF with 2 particles NF observation likelihood against true state evolution given the same control inputs for 100 dimensions.