Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Policy-Based RL Algorithms for POMDPs

Ameya Anjarlekar, Rasoul Etesami, R Srikant

TL;DR

This work addresses learning policies for POMDPs with continuous belief states by transforming the problem into a finite-state Superstate MDP through truncation of observation-action histories to length $l$. It establishes a rigorous bound showing the optimal value of the Superstate MDP closely tracks the POMDP optimal value, with an exponentially decaying gap in $l$, under a Uniform Filter Stability condition. Building on this, the authors develop a policy-based learning framework using TD-learning with linear function approximation to learn the Superstate Q-function, followed by a POLITEX-style policy update, and provide finite-time regret bounds that separate errors from function approximation and history truncation. The approach yields scalable, theoretically justified PORL for POMDPs, and extends to discounted rewards, offering practical guarantees for learning in non-Markovian settings. Overall, the paper contributes tighter approximation bounds, a computationally efficient TD-based learning scheme, and provable convergence guarantees for policy optimization on finite-history abstractions of POMDPs.

Abstract

The continuous nature of belief states in POMDPs presents significant computational challenges in learning the optimal policy. In this paper, we consider an approach that solves a Partially Observable Reinforcement Learning (PORL) problem by approximating the corresponding POMDP model into a finite-state Markov Decision Process (MDP) (called Superstate MDP). We first derive theoretical guarantees that improve upon prior work that relate the optimal value function of the transformed Superstate MDP to the optimal value function of the original POMDP. Next, we propose a policy-based learning approach with linear function approximation to learn the optimal policy for the Superstate MDP. Consequently, our approach shows that a POMDP can be approximately solved using TD-learning followed by Policy Optimization by treating it as an MDP, where the MDP state corresponds to a finite history. We show that the approximation error decreases exponentially with the length of this history. To the best of our knowledge, our finite-time bounds are the first to explicitly quantify the error introduced when applying standard TD learning to a setting where the true dynamics are not Markovian.

Scalable Policy-Based RL Algorithms for POMDPs

TL;DR

This work addresses learning policies for POMDPs with continuous belief states by transforming the problem into a finite-state Superstate MDP through truncation of observation-action histories to length . It establishes a rigorous bound showing the optimal value of the Superstate MDP closely tracks the POMDP optimal value, with an exponentially decaying gap in , under a Uniform Filter Stability condition. Building on this, the authors develop a policy-based learning framework using TD-learning with linear function approximation to learn the Superstate Q-function, followed by a POLITEX-style policy update, and provide finite-time regret bounds that separate errors from function approximation and history truncation. The approach yields scalable, theoretically justified PORL for POMDPs, and extends to discounted rewards, offering practical guarantees for learning in non-Markovian settings. Overall, the paper contributes tighter approximation bounds, a computationally efficient TD-based learning scheme, and provable convergence guarantees for policy optimization on finite-history abstractions of POMDPs.

Abstract

The continuous nature of belief states in POMDPs presents significant computational challenges in learning the optimal policy. In this paper, we consider an approach that solves a Partially Observable Reinforcement Learning (PORL) problem by approximating the corresponding POMDP model into a finite-state Markov Decision Process (MDP) (called Superstate MDP). We first derive theoretical guarantees that improve upon prior work that relate the optimal value function of the transformed Superstate MDP to the optimal value function of the original POMDP. Next, we propose a policy-based learning approach with linear function approximation to learn the optimal policy for the Superstate MDP. Consequently, our approach shows that a POMDP can be approximately solved using TD-learning followed by Policy Optimization by treating it as an MDP, where the MDP state corresponds to a finite history. We show that the approximation error decreases exponentially with the length of this history. To the best of our knowledge, our finite-time bounds are the first to explicitly quantify the error introduced when applying standard TD learning to a setting where the true dynamics are not Markovian.

Paper Structure

This paper contains 31 sections, 11 theorems, 77 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Description and Prior Results
  4. POMDP Structure and Notations
  5. Belief State MDP Representation of the POMDP
  6. Bellman's Optimality and Uniqueness of Solution
  7. Algorithm and Theoretical Results
  8. Defining an Approximate MDP Using Finite Observations
  9. How good is the approximate MDP?
  10. Intuition about Assumption \ref{['as:UFSC']}
  11. An Approximate Policy Optimization Algorithm to Learn the Optimal Policy corresponding to the Superstate MDP
  12. Conclusion
  13. Appendix A: Omitted Proofs
  14. Proof of Lemma \ref{['lemma: belief approx']}
  15. Proof of Lemma \ref{['lemma: norm bound']}
  16. ...and 16 more sections

Key Result

Theorem 1

Consider an infinite horizon discounted reward POMDP with discount factor $\gamma \in [0,1)$, and finite state and action spaces $\mathcal{S}$ and $\mathcal{A}$, respectively. Then,

Figures (3)

  • Figure 1: T-Maze
  • Figure 2: Average reward per episode for varying history lengths $k$ and observation noise levels $p$.
  • Figure 3: Comparing Moving Average Reward

Theorems & Definitions (13)

  • Theorem 1
  • Definition 1
  • Lemma 1
  • Lemma 2
  • Theorem 2
  • Lemma 3
  • Lemma 4
  • Theorem 3
  • Corollary 1
  • Lemma 5
  • ...and 3 more