Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs

Abhishek Gupta, Aditya Mahajan

Abstract

Markov decision processes (MDPs) is viewed as an optimization of an objective function over certain linear operators over general function spaces. Using the well-established perturbation theory of linear operators, this viewpoint allows one to identify derivatives of the objective function as a function of the linear operators. This leads to generalization of many well-known results in reinforcement learning to cases with generate state and action spaces. Prior results of this type were only established in the finite-state finite-action MDP settings and in settings with certain linear function approximations. The framework also leads to new low-complexity PPO-type reinforcement learning algorithms for general state and action space MDPs.

Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs

Abstract

Markov decision processes (MDPs) is viewed as an optimization of an objective function over certain linear operators over general function spaces. Using the well-established perturbation theory of linear operators, this viewpoint allows one to identify derivatives of the objective function as a function of the linear operators. This leads to generalization of many well-known results in reinforcement learning to cases with generate state and action spaces. Prior results of this type were only established in the finite-state finite-action MDP settings and in settings with certain linear function approximations. The framework also leads to new low-complexity PPO-type reinforcement learning algorithms for general state and action space MDPs.
Paper Structure (46 sections, 19 theorems, 44 equations, 1 figure)

This paper contains 46 sections, 19 theorems, 44 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Motivating Applications
  3. Active Flow Control
  4. Mean Field Control (MFC)
  5. Soft Robotics Control Design
  6. Contributions
  7. Notation
  8. Technical background
  9. Primer on Linear Operators
  10. Operator Norm
  11. Compact Operator
  12. Spectral Radius:
  13. Neumann Series
  14. Inf-compact functions
  15. Integral Probability Metrics
  16. ...and 31 more sections

Key Result

Lemma 2.1

Let $\bm A,\bm B\in\mathcal{B}(\mathcal{X},\mathcal{X})$ be two bounded linear operators such that $\bm A^{-1}$ is also bounded. Then there exists $\bar{\epsilon}>0$ such that for every $\epsilon\in[-\bar{\epsilon},\bar{\epsilon}]$, $\bm A+\epsilon \bm B$ is an invertible operator with a bounded in

Figures (1)

  • Figure 1: Simulation of MM-RKHS and PPO algorithms. The left figure contains the simulation results for the deterministic algorithm, which requires precise knowledge of transition kernel and cost function. The right figure simulates the two algorithms using sampled trajectories, wherein the cost function and transition structure generates multiple trajectories, which are used to update the policy.

Theorems & Definitions (55)

  • Lemma 2.1
  • Proof 1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Example 1: Linear Quadratic Regulator (LQR)
  • Remark 3.1
  • Remark 3.2
  • Definition 3.3: Spectrally Stable Policies
  • Definition 3.4: Decaying Cost Policies
  • ...and 45 more