Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions

Jingyang You, Hanna Kurniawati

TL;DR

The paper tackles generalisation in reinforcement learning under uncertain dynamics by formulating Bayes-Adaptive MDPs with learnable models. It presents GLiBRL, a Generalised Linear Model-based BRL method that uses learnable basis functions to achieve exact posterior updates and a closed-form marginal likelihood, avoiding the difficulties of ELBO optimisation. Empirically, GLiBRL substantially improves zero-shot performance and exhibits low variance on challenging MetaWorld ML10/ML45 benchmarks, outperforming VariBAD, MAML, RL$^2$, and other baselines. The work highlights the potential of ELBO-free Bayesian inference in BRL and points to future directions in model-based planning and efficient online belief updates.

Abstract

Bayesian Reinforcement Learning (BRL) provides a framework for generalisation of Reinforcement Learning (RL) problems from its use of Bayesian task parameters in the transition and reward models. However, classical BRL methods assume known forms of transition and reward models, reducing their applicability in real-world problems. As a result, recent deep BRL methods have started to incorporate model learning, though the use of neural networks directly on the joint data and task parameters requires optimising the Evidence Lower Bound (ELBO). ELBOs are difficult to optimise and may result in indistinctive task parameters, hence compromised BRL policies. To this end, we introduce a novel deep BRL method, Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions (GLiBRL), that enables efficient and accurate learning of transition and reward models, with fully tractable marginal likelihood and Bayesian inference on task parameters and model noises. On challenging MetaWorld ML10/45 benchmarks, GLiBRL improves the success rate of one of the state-of-the-art deep BRL methods, VariBAD, by up to 2.7x. Comparing against representative or recent deep BRL / Meta-RL methods, such as MAML, RL2, SDVT, TrMRL and ECET, GLiBRL also demonstrates its low-variance and decent performance consistently.

Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions

TL;DR

The paper tackles generalisation in reinforcement learning under uncertain dynamics by formulating Bayes-Adaptive MDPs with learnable models. It presents GLiBRL, a Generalised Linear Model-based BRL method that uses learnable basis functions to achieve exact posterior updates and a closed-form marginal likelihood, avoiding the difficulties of ELBO optimisation. Empirically, GLiBRL substantially improves zero-shot performance and exhibits low variance on challenging MetaWorld ML10/ML45 benchmarks, outperforming VariBAD, MAML, RL, and other baselines. The work highlights the potential of ELBO-free Bayesian inference in BRL and points to future directions in model-based planning and efficient online belief updates.

Abstract

Bayesian Reinforcement Learning (BRL) provides a framework for generalisation of Reinforcement Learning (RL) problems from its use of Bayesian task parameters in the transition and reward models. However, classical BRL methods assume known forms of transition and reward models, reducing their applicability in real-world problems. As a result, recent deep BRL methods have started to incorporate model learning, though the use of neural networks directly on the joint data and task parameters requires optimising the Evidence Lower Bound (ELBO). ELBOs are difficult to optimise and may result in indistinctive task parameters, hence compromised BRL policies. To this end, we introduce a novel deep BRL method, Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions (GLiBRL), that enables efficient and accurate learning of transition and reward models, with fully tractable marginal likelihood and Bayesian inference on task parameters and model noises. On challenging MetaWorld ML10/45 benchmarks, GLiBRL improves the success rate of one of the state-of-the-art deep BRL methods, VariBAD, by up to 2.7x. Comparing against representative or recent deep BRL / Meta-RL methods, such as MAML, RL2, SDVT, TrMRL and ECET, GLiBRL also demonstrates its low-variance and decent performance consistently.
Paper Structure (25 sections, 41 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 25 sections, 41 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Processes and Reinforcement Learning
  4. Bayes-Adaptive MDPs and Bayesian Reinforcement Learning
  5. Generalised Linear Models in Bayesian RL with Learnable Basis Functions
  6. Learning the Forms of Transitions and Rewards
  7. Learning the Policy
  8. Related Work
  9. Experiments
  10. Implementation Details
  11. Comparisons among GLiBRL, VariBAD, MAML and RL$^2$
  12. Comparisons among GLiBRL, SDVT, TrMRL and ECET
  13. Ablation Studies
  14. Conclusion
  15. Appendix
  16. ...and 10 more sections

Figures (4)

  • Figure 1: IQM and $95\%$ CI of testing success rate of GLiBRL, VariBAD, MAML and RL$^2$, with related to the number of training steps. Left: the ML10 benchmark; Right: the ML45 benchmark.
  • Figure 2: IQM and $95\%$ CI of errors in transition and reward predictions, comparing GLiBRL and GLiBRL_wo_NI. Up: Transitions; Bottom: Rewards; Left: ML10; Right: ML45.
  • Figure 3: IQM and $95\%$ CI of success rate for each testing scenario in ML10. The Shelf Place scenario is challenging as none of the method can achieve a single success.
  • Figure 4: IQM and $95\%$ CI of success rate for each testing scenario in ML45. GLiBRL achieves nearly $100\%$ testing success rates in both Door Lock and Door Unlock scenarios.