Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

Abdullah Akgül, Manuel Haußmann, Melih Kandemir

TL;DR

Offline model-based RL suffers from distributional shift and overestimation when using MC-based Bellman targets, leading to slow convergence. The authors introduce MOMBO, a deterministic uncertainty propagation method that uses progressive moment matching to pass next-state uncertainty through a Q-network, yielding a lower confidence Bellman target and avoiding MC sampling. They establish nonasymptotic $1$-Wasserstein bounds and tighter suboptimality guarantees for MOMBO compared with sampling-based PEVI, and demonstrate faster convergence with strong final performance on D4RL, including mixed datasets with limited expert data. Overall, MOMBO provides a theoretically principled and practically effective approach to safe, sample-efficient offline reinforcement learning by eliminating stochastic Bellman targets while preserving uncertainty information.

Abstract

Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference-based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

TL;DR

Offline model-based RL suffers from distributional shift and overestimation when using MC-based Bellman targets, leading to slow convergence. The authors introduce MOMBO, a deterministic uncertainty propagation method that uses progressive moment matching to pass next-state uncertainty through a Q-network, yielding a lower confidence Bellman target and avoiding MC sampling. They establish nonasymptotic -Wasserstein bounds and tighter suboptimality guarantees for MOMBO compared with sampling-based PEVI, and demonstrate faster convergence with strong final performance on D4RL, including mixed datasets with limited expert data. Overall, MOMBO provides a theoretically principled and practically effective approach to safe, sample-efficient offline reinforcement learning by eliminating stochastic Bellman targets while preserving uncertainty information.

Abstract

Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference-based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.
Paper Structure (42 sections, 11 theorems, 72 equations, 6 figures, 5 tables, 1 algorithm)

This paper contains 42 sections, 11 theorems, 72 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement learning.
  4. Offline reinforcement learning
  5. Scope and problem statement.
  6. Bellman operators.
  7. Pessimistic value iteration algorithms
  8. Theoretical analysis of sampling-based PEVI
  9. MOMBO: Moment matching offline model-based policy optimization
  10. Theoretical analysis of moment matching-based uncertainty propagation
  11. Implementation details of MOMBO
  12. Experiments
  13. Conclusion
  14. Limitations.
  15. Related work
  16. ...and 27 more sections

Key Result

Theorem 1

For any $\pi$ derived with $\mathbb{A}_{\texttt{PEVI}}(\BQpessimisticsample, \widehat{\mathrm{P}})$ that satisfies with probability at least $1 - \delta$ for some error tolerance $\delta \in (0, 1)$ where $s' \sim \mathrm{P}(\cdot| s, a)$, the following inequality holds for any initial state $s_1 \in \mathcal{S}$:

Figures (6)

  • Figure 1: Moment Matching versus Monte Carlo Sampling. Moment matching offers sharp estimates of the action-value of the next state at the cost of only two forward passes through a critic network. A similar sharpness cannot be reached even with $10000$ Monte Carlo samples, which is $5000$ times more costly. See \ref{['appsec:mm_vs_sampling']} for details.
  • Figure 2: Evaluation results on halfcheetah for four settings. The dashed, dotted, and solid curves represent the mean of the normalized rewards across ten evaluation episodes and four random seeds. The shaded area indicates one standard deviation from the mean.
  • Figure 3: Moment Matching versus Monte Carlo Sampling. A comparison of moment matching and Monte Carlo sampling methods for estimating the next value for all tasks in the D4RL dataset.
  • Figure 4: Learning curves for the D4RL dataset.
  • Figure 5: Learning curves for the mixed dataset with trained policy demonstration ratios of $\texttt{0.01}$ and 0.05.
  • ...and 1 more figures

Theorems & Definitions (22)

  • Theorem 1: Suboptimality of PEVI jin2021pessimism
  • Theorem 2: Suboptimality of sampling-based PEVI
  • Lemma 1: Moment matching
  • Lemma 2: Moment matching bound
  • Lemma 3: Moment matching MLP bound
  • Theorem 3: Suboptimality of moment matching-based PEVI algorithms
  • proof : Proof of \ref{['lem:mmproperties']}.
  • Lemma 4: An inequality between normal cdfs
  • proof : Proof of \ref{['lem:cdfinequality']}.
  • Definition 1: Wasserstein distance
  • ...and 12 more