FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning

TL;DR

The paper introduces Fourier Latent Dynamics (FLD), a self-supervised, structured representation that enforces latent dynamics in a continuously parameterized space to model periodic and quasi-periodic motions. Building on Periodic Autoencoder (PAE), FLD propagates latent states with globally constant parameters $(f,a,b)$ and uses a multi-step loss $L_{FLD}^N=\sum_{i=0}^N \alpha^i L_i$ to achieve stable long-horizon prediction, enabling autoregressive motion synthesis. In the motion-learning pipeline, FLD supplies targets via a learned latent space $\Theta$ and a local phase $\phi_t$, guiding a policy trained with PPO; online tracking includes a threshold-based evaluation and a fallback mechanism to safe states when proposals deviate from learned dynamics. Through experiments on a MIT Humanoid platform with locomotion data, FLD demonstrates a more structured latent manifold, improved long-horizon reconstruction/prediction, and robust online tracking with adaptive skill samplers (e.g., ALPGMM) that enhance generalization while avoiding unlearnable regions. Overall, FLD offers a principled approach to open-ended motion learning by exploiting spatial-temporal structure in the latent space to interpolate, generate, and safely track novel motions.

Abstract

Motion trajectories offer reliable references for physics-based motion learning but suffer from sparsity, particularly in regions that lack sufficient data coverage. To address this challenge, we introduce a self-supervised, structured representation and generation method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. The motion dynamics in a continuously parameterized latent space enable our method to enhance the interpolation and generalization capabilities of motion learning algorithms. The motion learning controller, informed by the motion parameterization, operates online tracking of a wide range of motions, including targets unseen during training. With a fallback mechanism, the controller dynamically adapts its tracking strategy and automatically resorts to safe action execution when a potentially risky target is proposed. By leveraging the identified spatial-temporal structure, our work opens new possibilities for future advancements in general motion representation and learning algorithms.

Figures (17)

• Figure 1: FLD training pipeline. During training, latent dynamics are enforced to predict proceeding latent states and parameterizations. The prediction loss is computed in the original motion space with respect to the ground truth future states.
• Figure 2: System overview. During training, the latent states propagate under the latent dynamics and are reconstructed to policy tracking targets $\hat{s}$ at each step. The tracking reward $r^T$ is computed as the distance between the target $\hat{s}$ and the measured states $s$.
• Figure 3: Online tracking and fallback mechanism. (a) The prediction loss $L_{FLD\xspace}^N$ is evaluated within an input buffer of user-proposed tracking targets. The mechanism accepts the proposal only when the prediction loss is below a threshold $\epsilon_{FLD\xspace}$. (b) The proposed tracking targets (dashed curve) may contain risky states (dashed red dots). The fallback mechanism identifies these states and defaults them to safe alternatives (dashed red arrows and green dots) by propagating latent dynamics. Note that the real-time tracking trajectories are not necessarily periodic or quasi-periodic.
• Figure 4: Latent manifolds for different motions. Each color is associated with a trajectory from a motion type. The arrows denote the state evolution direction. FLD presents the strongest spatial-temporal relationships with explicit latent dynamics enforcement. PAE witnesses a similar but weaker pattern with local sinusoidal reconstruction. In comparison, VAE enables only spatial closeness, and the trajectories of the original states are the least structured.
• Figure 5: Motion reconstruction and prediction of a diagonal run trajectory (left). The solid and dashed curves denote the ground truth and predicted state evolution. The relative prediction error (vivid) of FF, PAE and FLD is depicted with the axis indicating $e$ on the right. Latent offset (right) of step in place, forward run, and forward stride. Each radius denotes a latent channel.