On Foundation Models for Dynamical Systems from Purely Synthetic Data

TL;DR

The paper investigates whether foundation-models can perform low-level state prediction for dynamical systems by pretraining a decoder-only transformer on purely synthetic RKHS-generated trajectories. The proposed TimesFM-style architecture generalizes to unseen systems in both simulation and hardware (cart-pole and Furuta pendulum) and can be effectively fine-tuned with limited data to improve performance. Key contributions include a scalable RKHS-based data generation pipeline, an autoregressive transformer tailored for time-series dynamics, and a thorough evaluation of generalization, data efficiency, and robustness. The results indicate that synthetic-data pretraining can yield a robust, data-efficient foundation for real-time control, with practical implications for deploying generalist models across diverse dynamical domains.

Abstract

Foundation models have demonstrated remarkable generalization, data efficiency, and robustness properties across various domains. In this paper, we explore the feasibility of foundation models for applications in the control domain. The success of these models is enabled by large-scale pretaining on Internet-scale datasets. These are available in fields like natural language processing and computer vision, but do not exist for dynamical systems. We address this challenge by pretraining a transformer-based foundation model exclusively on synthetic data and propose to sample dynamics functions from a reproducing kernel Hilbert space. Our pretrained model generalizes for prediction tasks across different dynamical systems, which we validate in simulation and hardware experiments, including cart-pole and Furuta pendulum setups. Additionally, the model can be fine-tuned effectively to new systems to increase performance even further. Our results demonstrate the feasibility of foundation models for dynamical systems that outperform specialist models in terms of generalization, data efficiency, and robustness.

Figures (6)

• Figure 1: Proposed approach for a foundation model that predicts future states of dynamical systems. The model is pretrained purely on synthetic data (green). We propose to sample dynamics functions from an RKHS $\mathcal{H}$ and creating trajectory data by evolving the system through time. The foundation model is capable to zero-shot unseen systems in simulation and hardware (blue). Further, it can quickly be fine-tuned (dashed) to specific systems to improve the performance even further.
• Figure 2: Representative functions sampled from an RKHS (left) in a two-dimensional state ($x_1, x_2$) space and an example trajectory generated from the sampled dynamics functions (right).
• Figure 3: Architecture of the decoder-only transformer model. The model projects inputs into an embedding space using a residual block (RB), followed by $N$ causal decoder blocks with multi-head self-attention (MHA) and feed-forward networks (FFN). A final residual block projects from the embedding space to the output space.
• Figure 4: Custom cart-pole (image taken from hose2024parameteradaptiveapproximatempctuning) and Quanser Qube Servo 2 Furuta systems.
• Figure 5: Simulation results for cart-pole with fixed parameters and constant action (left) and randomized parameters with pink noise action (middle). We compare the MSE of our models (Pre, Ft) against baselines (LR, FNN, ST) across data subsets, showing generalization, data efficiency, and robustness (range of MSE values). The right shows an example trajectory, comparing the pretrained and fine-tuned models with the actual elocities $\dot{x}$ and $\dot{\theta}$ (note the different scaling of y-axes).