Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance of NPG in Countable State-Space Average-Cost RL

Yashaswini Murthy, Isaac Grosof, Siva Theja Maguluri, R. Srikant

TL;DR

This work designs a state-dependent step-size rule which dramatically improves the performance of NPG and theoretically shows that the iteration complexity of NPG can be made independent of the size of the state space.

Abstract

We consider policy optimization methods in reinforcement learning settings where the state space is arbitrarily large, or even countably infinite. The motivation arises from control problems in communication networks, matching markets, and other queueing systems. Specifically, we consider the popular Natural Policy Gradient (NPG) algorithm, which has been studied in the past only under the assumption that the cost is bounded and the state space is finite, neither of which holds for the aforementioned control problems. Assuming a Lyapunov drift condition, which is naturally satisfied in some cases and can be satisfied in other cases at a small cost in performance, we design a state-dependent step-size rule which dramatically improves the performance of NPG for our intended applications. In addition to experimentally verifying the performance improvement, we also theoretically show that the iteration complexity of NPG can be made independent of the size of the state space. The key analytical tool we use is the connection between NPG step-sizes and the solution to Poisson's equation. In particular, we provide policy-independent bounds on the solution to Poisson's equation, which are then used to guide the choice of NPG step-sizes.

Performance of NPG in Countable State-Space Average-Cost RL

TL;DR

This work designs a state-dependent step-size rule which dramatically improves the performance of NPG and theoretically shows that the iteration complexity of NPG can be made independent of the size of the state space.

Abstract

We consider policy optimization methods in reinforcement learning settings where the state space is arbitrarily large, or even countably infinite. The motivation arises from control problems in communication networks, matching markets, and other queueing systems. Specifically, we consider the popular Natural Policy Gradient (NPG) algorithm, which has been studied in the past only under the assumption that the cost is bounded and the state space is finite, neither of which holds for the aforementioned control problems. Assuming a Lyapunov drift condition, which is naturally satisfied in some cases and can be satisfied in other cases at a small cost in performance, we design a state-dependent step-size rule which dramatically improves the performance of NPG for our intended applications. In addition to experimentally verifying the performance improvement, we also theoretically show that the iteration complexity of NPG can be made independent of the size of the state space. The key analytical tool we use is the connection between NPG step-sizes and the solution to Poisson's equation. In particular, we provide policy-independent bounds on the solution to Poisson's equation, which are then used to guide the choice of NPG step-sizes.
Paper Structure (32 sections, 12 theorems, 86 equations, 3 figures, 1 algorithm)

This paper contains 32 sections, 12 theorems, 86 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model and Preliminaries
  4. Natural Policy Gradient Algorithm
  5. Main Result and Discussion
  6. Main Result
  7. Discussion on Assumptions
  8. Finite but large state spaces
  9. Countable State Spaces with Abandonments
  10. Scheduling in Switches
  11. Policy Evaluation in Stochastic Networks with Countable States
  12. Proof outline and Key Insights
  13. Simulations
  14. Single Queue System
  15. Setting
  16. ...and 17 more sections

Key Result

Theorem 3.1

Consider the sequence of policies $\pi_1,\pi_2,\ldots,\pi_T$ obtained from Algorithm alg: NPG with a state-dependent step size $\eta_\mathbf{q} = \sqrt{\frac{8\log|\mathcal{A}|}{T}}\frac{1}{M_\mathbf{q}}$, where $M_\mathbf{q} = \left( 2\delta(\mathbf{q}) + \frac{2}{\epsilon}f^2(\mathbf{q})+\frac{4D} Then, under Assumptions asspn:asump1, asspn:asump2 and asspn:assump3, there exist constants $c', c"

Figures (3)

  • Figure 1: NPG in a single queue system
  • Figure 2: NPG in a two queue system
  • Figure 3: NPG with perfect policy evaluation

Theorems & Definitions (21)

  • Theorem 3.1
  • proof
  • Theorem 4.1
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • proof
  • Lemma A.4
  • proof
  • Lemma A.5
  • ...and 11 more