SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets

TL;DR

This work decomposes latent states into endogenous and exogenous parts via conservative sampling and estimating model uncertainty on the endogenous states only, and provides a theoretical guarantee of model uncertainty and performance bound of SeMOPO.

Abstract

Model-based offline reinforcement Learning (RL) is a promising approach that leverages existing data effectively in many real-world applications, especially those involving high-dimensional inputs like images and videos. To alleviate the distribution shift issue in offline RL, existing model-based methods heavily rely on the uncertainty of learned dynamics. However, the model uncertainty estimation becomes significantly biased when observations contain complex distractors with non-trivial dynamics. To address this challenge, we propose a new approach - \emph{Separated Model-based Offline Policy Optimization} (SeMOPO) - decomposing latent states into endogenous and exogenous parts via conservative sampling and estimating model uncertainty on the endogenous states only. We provide a theoretical guarantee of model uncertainty and performance bound of SeMOPO. To assess the efficacy, we construct the Low-Quality Vision Deep Data-Driven Datasets for RL (LQV-D4RL), where the data are collected by non-expert policy and the observations include moving distractors. Experimental results show that our method substantially outperforms all baseline methods, and further analytical experiments validate the critical designs in our method. The project website is \href{https://sites.google.com/view/semopo}{https://sites.google.com/view/semopo}.

Figures (10)

• Figure 1: The SeMOPO framework encompasses two parts: model learning and policy optimization. In the model learning phase, SeMOPO employs conservative sampling to select trajectories, which are then used to train models for endogenous and exogenous dynamics, each implemented as an ensemble of Gaussian distributions. During policy optimization, SeMOPO trains a policy $\pi_{\theta}(a_t|s^+_t)$ and a value model $V_{\theta}(s^+_t)$ based on the endogenous states generated by a sampled endogenous dynamics model $\tilde{T}^j$. SeMOPO uses the reward penalized by the variance of the endogenous dynamics models' predictions to train the value model.
• Figure 2: The model uncertainty estimation of SeMOPO and Offline DV2 on the LQV-D4RL dataset. We randomly select 1000 states and report the mean and standard deviation of uncertainty on them.
• Figure 3: Examples of uncertainty estimated by SeMOPO and Offline DV2. $U(s^+)$, $U(s^-)$, and $U(Z)$ represent uncertainty estimations for the endogenous state, the exogenous state, and the belief latent state, respectively.
• Figure 4: The performance comparison for ablated methods. Each row represents the comparative performance probabilities, complete with $95\%$ bootstrap confidence intervals, suggesting that Algorithm X is superior to Algorithm Y Agarwal2021DeepRL. These probabilities are derived from 50 runs of 4 seeds for every task to ensure robustness in the evaluation. We show the aggregated results for all nine tasks.
• Figure 5: Original observations (Obs.) and image reconstructions from the endogenous states (Eds.) of different ablated methods in the medium_replay and random datasets of the Cheetah Run task. SeMOPO (Se+CS) can preserve task-relevant information well, while others can not.

Theorems & Definitions (10)

• Definition 2.1
• Lemma 3.1
• Theorem 3.3
• Theorem 3.4
• Lemma 1.1
• proof
• Theorem 1.2
• proof
• Theorem 1.5
• proof