Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning in POMDP's via Direct Gradient Ascent

Jonathan Baxter, Peter L. Bartlett

TL;DR

The paper addresses learning optimal policies in partially observable environments by directly optimizing the average reward through policy gradients. It introduces GPOMDP, a gradient estimator that learns from a single trajectory with a discount factor β, and proves its convergence to a gradient-like quantity related to the true gradient as β → 1, with a bias–variance trade-off governed by the mixing time of the environment. To optimize, it couples GPOMDP with CONJPOMDP, a robust conjugate-gradient method that handles noisy gradient information via gradient-based line searches. Experiments on a toy three-state MDP illustrate the gradient estimates, the bias–variance dynamics, and the practical viability of converging to near-optimal policies. The work lays a foundation for scalable, gradient-based reinforcement learning in POMDPs without requiring state reconstruction or full knowledge of the environment.

Abstract

This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter $β\in [0,1)$, which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.

Reinforcement Learning in POMDP's via Direct Gradient Ascent

TL;DR

The paper addresses learning optimal policies in partially observable environments by directly optimizing the average reward through policy gradients. It introduces GPOMDP, a gradient estimator that learns from a single trajectory with a discount factor β, and proves its convergence to a gradient-like quantity related to the true gradient as β → 1, with a bias–variance trade-off governed by the mixing time of the environment. To optimize, it couples GPOMDP with CONJPOMDP, a robust conjugate-gradient method that handles noisy gradient information via gradient-based line searches. Experiments on a toy three-state MDP illustrate the gradient estimates, the bias–variance dynamics, and the practical viability of converging to near-optimal policies. The work lays a foundation for scalable, gradient-based reinforcement learning in POMDPs without requiring state reconstruction or full knowledge of the environment.

Abstract

This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter , which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.

Paper Structure

This paper contains 12 sections, 4 theorems, 14 equations, 3 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Mathematical Framework
  4. Gradient Ascent on $\hbox{\boldmath $\bf\eta(\theta)$}$
  5. Approximating the Gradient $\hbox{\boldmath $\bf\nabla \eta(\theta)$}$
  6. Estimating $\hbox{\boldmath $\bf{\nabla_{\!\!\beta}}\eta(\theta)$}$
  7. Stochastic Gradient Ascent Algorithms
  8. Experiments
  9. Training a Controller
  10. Gradient Estimates
  11. Training via Conjugate-Gradient Ascent
  12. Conclusion

Key Result

Theorem 1

For all $\theta\in{\mathbb R}^K$ and $\beta \in [0,1)$,

Figures (3)

  • Figure 1: A plot of $\frac{\|\nabla \eta - \Delta_T\|}{\|\nabla \eta\|}$ for the three-state Markov chain, for two values of the discount parameter $\beta$.
  • Figure 2: Graph showing the final bias in the estimate $\Delta_T$ (as measured by $\frac{\|\nabla \eta - \Delta_T\|}{\|\nabla \eta\|}$) as a function of $\beta$ for the three-state Markov chain. $\Delta_T$ was generated by Algorithm \ref{['algorithm:pgradmdp']}. Note both axes are log scales.
  • Figure 3: Performance of the three-state Markov chain controller trained by $\mathop{\mathrm{CONJPOMDP}}\nolimits$ as a function of the total number of iterations of the Markov chain.

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4