Investigating Compounding Prediction Errors in Learned Dynamics Models

TL;DR

This paper investigates the problem of compounding prediction errors in learned one-step dynamics models used for planning in model-based reinforcement learning. Through systematic experiments across underlying dynamics, process noise, state dimensionality, and data distribution, it identifies underlying system dynamics as the primary driver of long-horizon prediction errors, especially when poles approach instability. It demonstrates that error behavior can level off for many stable systems and highlights the limited reliability of complex model choices across different tasks, while simple models can be surprisingly competitive. The findings offer practical guidance for data collection, model design, and the move beyond purely one-step models to improve control in the presence of compounding errors.

Abstract

Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.

Figures (19)

• Figure 1: Showing the compounding errors, formally the per-step MSE (median, $65^\text{th}$, and $95^\text{th}$ percentiles), of state-space systems shown in Eq. \ref{['eq:statespace_mat']} with different poles, $\rho$. Compounding errors vary substantially with the underlying poles of the environment and diverge when the poles approach instability. All models are trained and evaluated on separate datasets of 100 trajectories.
• Figure 2: Comparing the MSE of prediction error per-step (median, $65^\text{th}$, and $95^\text{th}$ percentiles) on common model types and parametrizations on simulated robotic tasks of different dynamics, simulators, and dimension. There is a trend of error of predictions increasing with the task difficulty, but there is high variability in the performance of any one model type when comparing across platforms. All model types are trained and evaluated on the same datasets, maintaining separate datasets of 100 trajectories for test-train split.
• Figure 3: Comparing prediction accuracy when increasing the levels of process noise in the system above and below the default of $\omega_t \sim \mathcal{U}(-0.01,0.01)$ on all dimensions (median, $65^\text{th}$, and $95^\text{th}$ percentiles). The error between a system with default (shown in Fig. \ref{['fig:compound']}) and zero process noise shows that the default noise from the random action matrices determines the resulting prediction accuracy. An interesting feature is that when increasing the process noise from $10\times$ to $15\times$, the modelling accuracy degrades by a factor of 15.
• Figure 4: Showing how the randomly sampled input matrices, $\boldsymbol{B}$, and actions affect the per-step MSE with different eigenvalues (median, $65^\text{th}$, and $95^\text{th}$ percentiles) by collecting new data and evaluate newly trained models with $\boldsymbol{B}=0$. The random actions are the second leading cause of prediction error behind the unstable eigenvalues.
• Figure 5: Comparing compounding error with different state dimensions to see if input-output prediction size challenges the models when the underlying dynamics are regularized (shown is the MSE with median, $65^\text{th}$, and $95^\text{th}$ percentiles). State size does not have a substantial effect on the modeling error (the decreased variance of the error at each step in the state-sizes could be due to averaging over more state dimensions).

Theorems & Definitions (6)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5
• Remark 6