Reconstructing dynamics from sparse observations with no training on target system

TL;DR

The proposed hybrid machine-learning framework provides a paradigm of reconstructing complex and nonlinear dynamics in the extreme situation where training data does not exist and the observations are random and sparse.

Abstract

In applications, an anticipated situation is where the system of interest has never been encountered before and sparse observations can be made only once. Can the dynamics be faithfully reconstructed from the limited observations without any training data? This problem defies any known traditional methods of nonlinear time-series analysis as well as existing machine-learning methods that typically require extensive data from the target system for training. We address this challenge by developing a hybrid transformer and reservoir-computing machine-learning scheme. The key idea is that, for a complex and nonlinear target system, the training of the transformer can be conducted not using any data from the target system, but with essentially unlimited synthetic data from known chaotic systems. The trained transformer is then tested with the sparse data from the target system. The output of the transformer is further fed into a reservoir computer for predicting the long-term dynamics or the attractor of the target system. The power of the proposed hybrid machine-learning framework is demonstrated using a large number of prototypical nonlinear dynamical systems, with high reconstruction accuracy even when the available data is only 20% of that required to faithfully represent the dynamical behavior of the underlying system. The framework provides a paradigm of reconstructing complex and nonlinear dynamics in the extreme situation where training data does not exist and the observations are random and sparse.

Figures (21)

• Figure 1: Dynamics reconstruction from random and sparse data. (a) The textbook case of a random time series sampled at a frequency higher than the Nyquist frequency, where the dynamical data can be faithfully reconstructed. (b) Training data from the target system (left) and a segment of time series of six data points in a time interval containing approximately two cycles of oscillation. According to the Nyquist criterion, the signal can be faithfully reconstructed with more than 20 uniformly sampled data points (see text). When the data points are far fewer than 20 and even worse, they are randomly sampled, reconstruction becomes challenging. However, if training data from the same target system are available, existing machine-learning methods can be used to reconstruct the dynamics from the sparse data yeo2019data. (c) If no training data from the target system are available, hybrid machine learning proposed here provides a viable solution to reconstructing the dynamics from sparse data. (d) Problem statement. Given random and sparse data, the goal is to reconstruct the dynamics of the target system governed by $d\mathbf{x}/dt=f(\mathbf{x},t)$. A hurdle that needs to be overcome is that, for any given three points, there exist infinitely many ways to fit the data, as illustrated on the right side. (e) Training of the machine-learning framework is done using complete data from a large number of synthetic dynamical systems $[\mathbf{h}_1,\mathbf{h}_2,\cdots,\mathbf{h}_k]$. The framework is then adapted to reconstruct and predict the dynamics of the target systems $[\mathbf{f}_1,\cdots,\mathbf{f}_m]$. (f) An example: in the testing (deployment) phase, sparse observations are provided to the trained neural network for dynamics reconstruction.
• Figure 2: Transformer architecture. The transformer receives the sparse and random observation as the input and generates the reconstructed output. See text for a detailed mathematical description.
• Figure 3: Illustration of the transformer-based dynamics reconstruction framework. (a) Training (adaptation) phase, where the model is trained on various synthetic chaotic systems, each divided into segments with randomly distributed sequence lengths $L_s$ and sparsity $S_r$. The data is masked before being input into the transformer, and the ground truth is used to minimize the MSE loss and smoothness loss with the output. By learning a randomly chosen segment from a random training system each time, the transformer is trained to handle data with varying lengths and different levels of sparsity. (b) Testing (deployment) phase. The testing systems are distinct from those in the training phase, i.e., the transformer is not trained on any of the testing systems. Given sparsely observed set of points, the transformer is able to reconstruct the dynamical trajectory.
• Figure 4: Performance of dynamics reconstruction. (a) Illustration of reconstruction results for the chaotic food-chain and Lotka-Volterra systems as the testing targets that the transformer has never been exposed to. For each target system, two sets of sparse measurements of different length $L_s$ and sparsity $S_r$ are shown. The trained transformer reconstructs the complete time series in each case. (b) Color-coded ensemble-averaged MSE values in the parameter plane $(L_s,S_r)$ (b1). Examples of testing MSE versus $S_r$ and $L_s$ only are shown in (b2) and (b3), respectively. (c) Ensemble-averaged reconstruction stability indicator $R_s({\rm MSE_c})$ versus $S_r$ and $L_s$, the threshold MSE is $\rm MSE_c=0.01$. (d) Robustness of dynamics reconstruction against noise: ensemble-averaged MSE in the parameter plane $(\sigma,S_r)$ (d1) and $(\sigma,L_s)$ (d2), with $\sigma$ being the noise amplitude. An example of reconstruction under noise of amplitude $\sigma=0.1$ is shown in (d3). The values of the performance indicators are the result of averaging over 50 independent statistical realizations.
• Figure 5: Reservoir-computing based long-term dynamics prediction. (a) An illustration of hybrid transformer/reservoir-computing framework. The time series reconstructed by the transformer is used to train the reservoir computer that generates time series of the target system of arbitrary length, leading to a reconstructed attractor that agrees with the ground truth. (b) RMSE and DV versus the sparsity parameter. (c) Color-coded ensemble-averaged DV in the reservoir-computing hyperparameter plane $(T_l,N_s)$ for $S_r=0.8$. (d) DV versus training length $T_l$ for $N_s = 500$ and versus reservoir network size $N_s$ for $T_l=10^5$. In all cases, 50 independent realizations are used.