Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

Yu-Wei Yang, Yun-Ming Chan, Wei Hung, Xi Liu, Ping-Chun Hsieh

TL;DR

This work addresses the challenge of selecting among learned dynamics models in offline model-based RL, where distribution shift undermines validation and off-policy evaluation. It introduces BOMS, a Bayesian optimization framework that treats model selection as a black-box optimization problem over candidate dynamics models, guided by a Gaussian process with a novel model-induced kernel. The kernel measures model similarity via one-step predictive differences and is backed by a theoretical bound on policy performance differences, enabling efficient online querying with a small budget. Empirically, BOMS yields substantial improvements over traditional model selection schemes across MuJoCo, Adroit, and Meta-World tasks, achieving near-best performance with only about $1 ext%$-$2.5 ext%$ online data, and it generalizes to augment other offline MBRL methods.

Abstract

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only $1\%$-$2.5\%$ of offline training data on various RL tasks.

Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

TL;DR

This work addresses the challenge of selecting among learned dynamics models in offline model-based RL, where distribution shift undermines validation and off-policy evaluation. It introduces BOMS, a Bayesian optimization framework that treats model selection as a black-box optimization problem over candidate dynamics models, guided by a Gaussian process with a novel model-induced kernel. The kernel measures model similarity via one-step predictive differences and is backed by a theoretical bound on policy performance differences, enabling efficient online querying with a small budget. Empirically, BOMS yields substantial improvements over traditional model selection schemes across MuJoCo, Adroit, and Meta-World tasks, achieving near-best performance with only about - online data, and it generalizes to augment other offline MBRL methods.

Abstract

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only - of offline training data on various RL tasks.

Paper Structure

This paper contains 18 sections, 3 theorems, 12 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Recasting Model Selection as BO
  5. Model-Induced Kernels for BOMS
  6. Experiments
  7. Experimental Setup
  8. Results and Discussions
  9. Related Work
  10. Conclusion
  11. Proof of Proposition \ref{['prop:model_dis']}
  12. Experimental Configurations
  13. Additional Related Work
  14. Additional Experimental Results
  15. Comparison of Model Selection Methods
  16. ...and 3 more sections

Key Result

Proposition 3.0

Given two policies $\widetilde{\pi}_1$ and $\widetilde{\pi}_2$ which are the optimal policies learned on models $\widetilde{M}_1=(P_1,r_1)$ and $\widetilde{M}_2=(P_2,r_2)$, respectively. Then, we have where the expectation is taken over $s\sim\omega$, $a\sim\widetilde{\pi}_1$, $s'_1\sim {P_1}(\cdot\rvert s,a)$, and $s'_2\sim {P_2}(\cdot \rvert s,a)$ and $\epsilon(\pi_\beta):= \mathbb{E}_{(s,a)\si

Figures (6)

  • Figure 1: An illustration of the BOMS framework.
  • Figure 2: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in all the tasks after 5 iterations, which correspond to only $1\%$-$2.5\%$ of the offline training data.
  • Figure 3: Comparison of various designs of model distance for BOMS in inference regret. These results corroborate the proposed model-induced kernel.
  • Figure 4: The inference regrets of BOMS with distance measure calculated under different rollout lengths.
  • Figure 5: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in almost all tasks after 5-10 iterations.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Proposition 3.0
  • Lemma A.1: Simulation Lemma
  • Proposition A.1
  • proof