Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Internal-State Policy-Gradient Methods for POMDPs

Douglas Aberdeen, Jonathan Baxter

TL;DR

The paper tackles learning policies with memory in partially observable settings by introducing three FSC-based policy-gradient algorithms. GAMP uses a known POMDP model to compute low-variance gradient estimates via a matrix-series expansion; IState-GPOMDP and Exp-GPOMDP provide model-free options with variance reduction, including a Rao-Blackwellised variant. Empirical results on large-scale POMDPs, including Heaven/Hell and Pentagon, demonstrate that memory-based approaches substantially outperform memoryless methods and that GAMP scales to thousands of states while model-free variants remain tractable with sparse FSCs. Overall, the work significantly advances scalable, memory-aware policy-gradient methods for large POMDPs and multi-agent domains.

Abstract

Policy-gradient methods have received increased attention recently as a mechanism for learning to act in partially observable environments. They have shown promise for problems admitting memoryless policies but have been less successful when memory is required. In this paper we develop several improved algorithms for learning policies with memory in an infinite-horizon setting -- directly when a known model of the environment is available, and via simulation otherwise. We compare these algorithms on some large POMDPs, including noisy robot navigation and multi-agent problems.

Scaling Internal-State Policy-Gradient Methods for POMDPs

TL;DR

The paper tackles learning policies with memory in partially observable settings by introducing three FSC-based policy-gradient algorithms. GAMP uses a known POMDP model to compute low-variance gradient estimates via a matrix-series expansion; IState-GPOMDP and Exp-GPOMDP provide model-free options with variance reduction, including a Rao-Blackwellised variant. Empirical results on large-scale POMDPs, including Heaven/Hell and Pentagon, demonstrate that memory-based approaches substantially outperform memoryless methods and that GAMP scales to thousands of states while model-free variants remain tractable with sparse FSCs. Overall, the work significantly advances scalable, memory-aware policy-gradient methods for large POMDPs and multi-agent domains.

Abstract

Policy-gradient methods have received increased attention recently as a mechanism for learning to act in partially observable environments. They have shown promise for problems admitting memoryless policies but have been less successful when memory is required. In this paper we develop several improved algorithms for learning policies with memory in an infinite-horizon setting -- directly when a known model of the environment is available, and via simulation otherwise. We compare these algorithms on some large POMDPs, including noisy robot navigation and multi-agent problems.

Paper Structure

This paper contains 14 sections, 4 theorems, 10 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. POMDPs with Internal-State
  3. Policy-Gradient Approaches
  4. Model-Based Policy-Gradient
  5. Asymptotic Convergence of GAMP
  6. A Multi-Agent Problem
  7. Model-less Policy-Gradient
  8. The IState-GPOMDP Algorithm
  9. Zero Gradient Regions
  10. Internal Belief States
  11. Empirical comparisons
  12. Heaven/Hell
  13. Pentagon
  14. Conclusion

Key Result

Theorem 1

Figures (6)

  • Figure 1: Evolution of a finite state POMDP controller.
  • Figure 2: Angular error between $\widehat{\nabla_N \eta}$ after $N$ iterations and the exact gradient for the Pentagon problem.
  • Figure 3: Plan of the factory floor with 13 locations. The dashed lines show an optimal policy.
  • Figure 4: The Heaven/Hell problem: the optimal policy is to visit the signpost to find out whether to move left or right at the top-middle state in order to reach heaven.
  • Figure 5: The Pentagon robot navigation problem maze.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4