Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Model-Based Reinforcement Learning with Anti-Exploration

Padmanaba Srinivasan, William Knottenbelt

TL;DR

MoMo addresses offline model-based reinforcement learning by extending the anti-exploration paradigm with Morse Neural Networks that serve as an implicit behavior policy and uncertainty estimator. It combines a policy constraint with an anti-exploration value bonus and introduces rollout truncation to stop synthetic trajectories that drift out of dataset support, enabling stable offline MBRL with a single dynamics model. Empirically, MoMo (both model-free and model-based variants) matches or exceeds baselines on numerous D4RL Locomotion and Adroit tasks, with the model-based version often delivering the strongest results and showing robustness to hyperparameters. The approach reduces reliance on ensemble uncertainty and provides calibrated, tractable uncertainty estimates, offering practical benefits for offline RL in data-constrained settings.

Abstract

Model-based reinforcement learning (MBRL) algorithms learn a dynamics model from collected data and apply it to generate synthetic trajectories to enable faster learning. This is an especially promising paradigm in offline reinforcement learning (RL) where data may be limited in quantity, in addition to being deficient in coverage and quality. Practical approaches to offline MBRL usually rely on ensembles of dynamics models to prevent exploitation of any individual model and to extract uncertainty estimates that penalize values in states far from the dataset support. Uncertainty estimates from ensembles can vary greatly in scale, making it challenging to generalize hyperparameters well across even similar tasks. In this paper, we present Morse Model-based offline RL (MoMo), which extends the anti-exploration paradigm found in offline model-free RL to the model-based space. We develop model-free and model-based variants of MoMo and show how the model-free version can be extended to detect and deal with out-of-distribution (OOD) states using explicit uncertainty estimation without the need for large ensembles. MoMo performs offline MBRL using an anti-exploration bonus to counteract value overestimation in combination with a policy constraint, as well as a truncation function to terminate synthetic rollouts that are excessively OOD. Experimentally, we find that both model-free and model-based MoMo perform well, and the latter outperforms prior model-based and model-free baselines on the majority of D4RL datasets tested.

Offline Model-Based Reinforcement Learning with Anti-Exploration

TL;DR

MoMo addresses offline model-based reinforcement learning by extending the anti-exploration paradigm with Morse Neural Networks that serve as an implicit behavior policy and uncertainty estimator. It combines a policy constraint with an anti-exploration value bonus and introduces rollout truncation to stop synthetic trajectories that drift out of dataset support, enabling stable offline MBRL with a single dynamics model. Empirically, MoMo (both model-free and model-based variants) matches or exceeds baselines on numerous D4RL Locomotion and Adroit tasks, with the model-based version often delivering the strongest results and showing robustness to hyperparameters. The approach reduces reliance on ensemble uncertainty and provides calibrated, tractable uncertainty estimates, offering practical benefits for offline RL in data-constrained settings.

Abstract

Model-based reinforcement learning (MBRL) algorithms learn a dynamics model from collected data and apply it to generate synthetic trajectories to enable faster learning. This is an especially promising paradigm in offline reinforcement learning (RL) where data may be limited in quantity, in addition to being deficient in coverage and quality. Practical approaches to offline MBRL usually rely on ensembles of dynamics models to prevent exploitation of any individual model and to extract uncertainty estimates that penalize values in states far from the dataset support. Uncertainty estimates from ensembles can vary greatly in scale, making it challenging to generalize hyperparameters well across even similar tasks. In this paper, we present Morse Model-based offline RL (MoMo), which extends the anti-exploration paradigm found in offline model-free RL to the model-based space. We develop model-free and model-based variants of MoMo and show how the model-free version can be extended to detect and deal with out-of-distribution (OOD) states using explicit uncertainty estimation without the need for large ensembles. MoMo performs offline MBRL using an anti-exploration bonus to counteract value overestimation in combination with a policy constraint, as well as a truncation function to terminate synthetic rollouts that are excessively OOD. Experimentally, we find that both model-free and model-based MoMo perform well, and the latter outperforms prior model-based and model-free baselines on the majority of D4RL datasets tested.
Paper Structure (14 sections, 1 theorem, 14 equations, 3 figures, 3 tables, 2 algorithms)

This paper contains 14 sections, 1 theorem, 14 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Related Work
  2. Preliminaries
  3. Morse Model-Based Offline RL
  4. Morse Neural Networks Learn an Implicit Behavior Policy
  5. Challenges when Constraining Offline MBRL
  6. Extending to Offline MBRL
  7. Practical Implementation
  8. Experiments
  9. D4RL Performance
  10. Ablating Threshold $\epsilon_{\text{trunc}}$
  11. Sensitivity to $\lambda$
  12. Importance of Anti-Exploration
  13. Discussion
  14. Conclusion

Key Result

Proposition 1

The Morse neural network is an EBM: ${E_{\psi}(x) = -\log M_{\psi(x)}}$.

Figures (3)

  • Figure 1: Unnormalized density plots from a Morse neural network with RBF kernel (Equation \ref{['eq: rbf kernel']}) for $\lambda = \{0.1, 1.0, 2.0, 4.0\}$. Eight action modes are uniformly spaced points on a unit circle with alternating modes assigned to each state. 64 action samples for each state are drawn with a standard deviation $0.01$ and clipped to lie in the range ${[-1.0, 1.0]}$. Subfig (a) displays the training data and actions sampled for each state. Color-coded training points are overlaid on density plots for convenience. The Morse network (2 hidden layers of size 64, ReLU nonlinearity, gradient penalty, and layer normalization) captures all modes in each state. Note that the density plots cover a larger action range of ${[-1.8, 1.8]}$.
  • Figure 2: Ablations of $\epsilon_{\text{trunc}}$ on -m-r and -m-e datasets. The standard deviation over seeds is also included for each ablation. Note that $\epsilon_{\text{trunc}} = 0.95$ is the configuration used for MoMo-mb for primary results and MoMo-mf scores are included here for reference.
  • Figure 3: Ablations of $\lambda$ on -m-r and -m-e datasets. We use $\lambda = 1.0$ for primary results and perform additional sweeps over ${\lambda = [0.1, 2.0, 4.0]}$. The standard deviation over seeds is omitted for the sake of clarity. For MoMo-mb we use clear bars and maintain $\epsilon_{\text{trunc}} = 0.95$ in experiments.

Theorems & Definitions (4)

  • Definition 1: Morse Kernel
  • Definition 2: Morse Neural Network
  • Proposition 1
  • Definition 3: OOD Truncation Function