Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning

Abdelhakim Benechehab, Albert Thomas, Balázs Kégl

TL;DR

This paper tackles offline reinforcement learning by challenging the common reliance on ensembles for uncertainty in model-based methods. It proposes a single, well-calibrated autoregressive dynamics model (DARMDN) and shows that it can outperform neural ensembles on static learning metrics and yield strong dynamic performance in Hopper/D4RL tasks. By decoupling model selection from agent training, the study isolates model quality and demonstrates that one-step calibration metrics (LR and KS) robustly predict downstream policy performance, even when long-horizon metrics decline due to error accumulation. The results highlight the practical value of calibrated uncertainty in conservative offline MBRL and motivate broader validation and a unified evaluation protocol for offline RL.

Abstract

We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. The standard solution is to rely on ensembles for uncertainty heuristics and to avoid exploiting the model where it is too uncertain. We challenge the popular belief that we must resort to ensembles by showing that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark. We also analyze static metrics of model-learning and conclude on the important model properties for the final performance of the agent.

Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning

TL;DR

This paper tackles offline reinforcement learning by challenging the common reliance on ensembles for uncertainty in model-based methods. It proposes a single, well-calibrated autoregressive dynamics model (DARMDN) and shows that it can outperform neural ensembles on static learning metrics and yield strong dynamic performance in Hopper/D4RL tasks. By decoupling model selection from agent training, the study isolates model quality and demonstrates that one-step calibration metrics (LR and KS) robustly predict downstream policy performance, even when long-horizon metrics decline due to error accumulation. The results highlight the practical value of calibrated uncertainty in conservative offline MBRL and motivate broader validation and a unified evaluation protocol for offline RL.

Abstract

We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. The standard solution is to rely on ensembles for uncertainty heuristics and to avoid exploiting the model where it is too uncertain. We challenge the popular belief that we must resort to ensembles by showing that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark. We also analyze static metrics of model-learning and conclude on the important model properties for the final performance of the agent.
Paper Structure (33 sections, 6 equations, 18 figures, 6 tables, 1 algorithm)

This paper contains 33 sections, 6 equations, 18 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. A baseline: model-based offline policy optimization (MOPO)
  5. Experimental setup
  6. Static metrics
  7. Dynamic metrics
  8. Experiments & Results
  9. One-step metrics (LR, OR, R2, and KS).
  10. Long-horizon metrics.
  11. Dynamic evaluation.
  12. Correlating static metrics and dynamic scores.
  13. D4RL benchmark.
  14. Conclusion
  15. Implementation details
  16. ...and 18 more sections

Figures (18)

  • Figure 1: Hopper
  • Figure 2: Histogram of Hopper's thigh ground truth quantiles, under the model distribution (D4RL medium dataset). The legend also includes the value of the KS calibratedness metric. The dotted red line indicates the ideal case when the quantiles follow a uniform distribution.
  • Figure 3: Long horizon explained variance R2($L$) in the D4RL random dataset.
  • Figure 4: The Spearman and Pearson correlations between the episodic return and LR/-KS metrics on the D4RL medium dataset.
  • Figure 5: Kolmogorov-Smirnov (KS) statistic (in red) of the predicted reward.
  • ...and 13 more figures