Combating the Compounding-Error Problem with a Multi-step Model

TL;DR

The paper tackles the compounding-error problem in model-based RL by introducing M^3, a multi-step transition model that directly predicts h-step outcomes and uses a fixed-start rollout to prevent feedback of noisy intermediate predictions. It provides theoretical value-function and generalization bounds showing decreased horizon-dependence and demonstrates empirically that M^3 improves both background and decision-time planning across multiple domains, reducing planning errors and improving sample efficiency. The work highlights computational considerations, discusses extensions to stochastic dynamics and ensembles, and outlines future directions for applying multi-step modeling to more complex domains. Collectively, the study argues that multi-step models offer a principled and practical path to more reliable model-based RL.

Abstract

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.

Figures (11)

• Figure 1: (top) a 3-step rollout using a one-step model. (bottom) a 3-step rollout using a multi-step model $\mathrm{M^3}$. Crucially, at each step of the multi-step rollout, the agent uses $s_1$ as the starting point. The output of each intermediate step is only used to compute the next action.
• Figure 2: A comparison of actor critic equipped with the learned models (Cart Pole, Acrobot, and Lunar Lander). We set the maximum look-ahead horizon $H=8$. Results are averaged over 100 runs, and higher is better. The multi-step model consistently matches or exceeds the one-step model.
• Figure 3: Area under the curve, which corresponds to average episode return, as a function of the look-ahead horizon $h$. Results for all three domains (Cart Pole, Acrobot, and Lunar Lander) are averaged over 100 runs. We add two additional baselines, namely the model-free critic, and a model-based critic trained with hallucination talvitie_hallucination_14venkatraman_multi_step
• Figure 4: Tree construction (left) and action-value estimation (right) strategies.
• Figure 5: A comparison between tree expansion and value-estimation strategies when using the one-step model for action selection (left). Comparison between the one-step model and $\mathrm{M^3}$ for action selection (right). x-axis denotes the $\widehat{Q}$ of agent at that episode, and y-axis denotes performance gain over model-free. Performance is defined as episode return averaged over 20 episodes. Note the inverted-U. Initially, $\widehat{Q}$ and the model are both bad, so model provides little benefit. Towards the end $\widehat{Q}$ gets better, so using the model is not beneficial. However, we get a clear benefit in the intermediate episodes because the model is faster to learn than $\widehat{Q}$.

Theorems & Definitions (18)

• Definition 1
• Definition 2
• Theorem 1
• Theorem 2
• Lemma 1
• Lemma 2
• Theorem 3
• Lemma 2
• proof
• Lemma 3