Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Connecting Stochastic Optimal Control and Reinforcement Learning

Jannes Quer, Enric Ribera Borrell

TL;DR

The main motivation is to apply importance sampling to sampling rare events which can be reformulated as an optimal control problem by using a parameterised approach and how the stochastic optimal control problem can be interpreted in the framework of reinforcement learning.

Abstract

In this paper the connection between stochastic optimal control and reinforcement learning is investigated. Our main motivation is to apply importance sampling to sampling rare events which can be reformulated as an optimal control problem. By using a parameterised approach the optimal control problem becomes a stochastic optimization problem which still raises some open questions regarding how to tackle the scalability to high-dimensional problems and how to deal with the intrinsic metastability of the system. To explore new methods we link the optimal control problem to reinforcement learning since both share the same underlying framework, namely a Markov Decision Process (MDP). For the optimal control problem we show how the MDP can be formulated. In addition we discuss how the stochastic optimal control problem can be interpreted in the framework of reinforcement learning. At the end of the article we present the application of two different reinforcement learning algorithms to the optimal control problem and a comparison of the advantages and disadvantages of the two algorithms.

Connecting Stochastic Optimal Control and Reinforcement Learning

TL;DR

The main motivation is to apply importance sampling to sampling rare events which can be reformulated as an optimal control problem by using a parameterised approach and how the stochastic optimal control problem can be interpreted in the framework of reinforcement learning.

Abstract

In this paper the connection between stochastic optimal control and reinforcement learning is investigated. Our main motivation is to apply importance sampling to sampling rare events which can be reformulated as an optimal control problem. By using a parameterised approach the optimal control problem becomes a stochastic optimization problem which still raises some open questions regarding how to tackle the scalability to high-dimensional problems and how to deal with the intrinsic metastability of the system. To explore new methods we link the optimal control problem to reinforcement learning since both share the same underlying framework, namely a Markov Decision Process (MDP). For the optimal control problem we show how the MDP can be formulated. In addition we discuss how the stochastic optimal control problem can be interpreted in the framework of reinforcement learning. At the end of the article we present the application of two different reinforcement learning algorithms to the optimal control problem and a comparison of the advantages and disadvantages of the two algorithms.
Paper Structure (25 sections, 57 equations, 7 figures, 3 algorithms)

This paper contains 25 sections, 57 equations, 7 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Importance sampling SOC problem
  3. The sampling problem
  4. Hamilton-Jacobi-Bellman equation
  5. Stochastic optimization problem
  6. Introduction to reinforcement learning
  7. Markov decision processes
  8. Reward and state-action transition probabilities
  9. Policies
  10. Trajectories, return and value functions
  11. RL as optimization problem
  12. Brief summary of RL algorithms
  13. Q-learning
  14. Policy gradient
  15. The SOC problem as RL formulation
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Trajectories following the not controlled policy for the two settings of study. The actions chosen along the trajectories are null. The trajectories are sampled starting at $s_\text{init} = -1$ until they arrive into the target set $\mathcal{T} = [1, \infty)$. Left panel: snapshots of the metastable trajectory $\beta=4$. Right panel: trajectory positions as a function of the time steps.
  • Figure 2: Left panel: estimation of $L^2(\mu_\theta)$ at each gradient step for the non-metastable setting $\beta=1$. Right panel: approximated policy for different gradient updates for the batch of trajectories case ($K=10^3$).
  • Figure 3: Left panel: estimation of $L^2(\mu_\theta)$ as a function of the gradient steps for the metastable setting $\beta=4$. Right panel: approximated policy for different gradient updates.
  • Figure 4: Left panel: estimation of $L^2(\mu_\theta)$ as a function of the trajectories for the non-metastable setting $\beta=1$. Right panel: approximated policy by the actor model after different trajectories.
  • Figure 5: Left panel: estimation of $L^2(\mu_\theta)$ as a function of the trajectories for the metastable setting $\beta=4$. Right panel: approximated policy by the actor model after different trajectories.
  • ...and 2 more figures