Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Partial Parametric Model Knowledge

Shuyuan Wang, Philip D. Loewen, Nathan P. Lawrence, Michael G. Forbes, R. Bhushan Gopaluni

TL;DR

This work adapts reinforcement learning methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment, using incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance.

Abstract

We adapt reinforcement learning (RL) methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment. Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes inspiration from both model-free RL and model-based control. It uses incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance. The linear quadratic regulator provides a case study; numerical experiments demonstrate the effectiveness and resulting benefits of the proposed method.

Reinforcement Learning with Partial Parametric Model Knowledge

TL;DR

This work adapts reinforcement learning methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment, using incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance.

Abstract

We adapt reinforcement learning (RL) methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment. Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes inspiration from both model-free RL and model-based control. It uses incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance. The linear quadratic regulator provides a case study; numerical experiments demonstrate the effectiveness and resulting benefits of the proposed method.
Paper Structure (14 sections, 32 equations, 4 figures, 1 algorithm)

This paper contains 14 sections, 32 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. The linear quadratic regulator (LQR)
  5. Policy iteration for LQR
  6. Problem Statement
  7. RL with Partial Knowledge
  8. Representing Partial Model Information
  9. Introducing Partial Model Information into RL
  10. Overall Algorithm
  11. Numerical Examples
  12. Example 1
  13. Example 2
  14. Conclusion

Figures (4)

  • Figure 1: The comparison between different methods in terms of the evolution of closed-loop system with respect to iterations. Each iteration contains 30 rollouts, with each rollout running 20 steps. The shaded area represents the variance of each method, with percentile 0-75%. The solid line represents the median of each method.
  • Figure 2: Stochastic trajectories generated from the learned controller. The $X$-axis represents simulation steps.
  • Figure 3: Results of PLSPI given a completely known system model, with percentile 0-100%.
  • Figure : Partial Knowledge LSPI (PLSPI)

Theorems & Definitions (1)

  • Remark 1