Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Point-Based Value Iteration via Active Sampling of Belief Points and Gaussian Process Regression

Siqiong Zhou, Ashif S. Iquebal, Esma S. Gel

TL;DR

The paper tackles the computational bottleneck of upper bound estimation in finite-horizon POMDPs solved via PBVI. It introduces GP-UCB, which uses Gaussian Process Regression trained on a strategically selected subset of informative belief points to predict the upper bound convex hull, reducing reliance on costly sawtooth projections. The authors prove PAC-like consistency and demonstrate substantial speedups (30-60% on small problems, up to 99.7% on larger ones) while maintaining the same bound gaps as traditional sawtooth methods. Experiments on five benchmark POMDPs show improved scalability and competitive accuracy, highlighting GP-UCB as a practical tool for large-scale, finite-horizon POMDP planning with uncertain observations.

Abstract

Partially Observable Markov Decision Processes (POMDPs) are fundamental to decision-making under uncertainty. We introduce a novel scalable approach to accelerate upper bound estimation in Point-Based Value Iteration (PBVI) algorithms, the leading method to solve large-scale POMDPs. PBVI approximates the value function using a set of belief points rather than the entire continuous belief space and relies on lower and upper bounds for convergence. While lower bounds are straightforward to compute, PVBI requires repeated sawtooth projection operations to approximate the upper bound convex hull, significantly increasing the computational burden although many of these sawtooth projections become redundant as the belief set expands. To address this, we infer the upper bound using the upper confidence bound of a Gaussian Process Regression (GP-UCB) fitted over a subset of the most informative reachable belief points--the ones that exhibit linear independence in some high-dimensional Hilbert space. This approach reduces the number of sawtooth projections by 84.3% on average without compromising the solution quality. We further establish the theoretical consistency of the proposed GP-UCB estimate of the upper bound and show convergence to the true upper bound convex hull. We implement GP-UCB and test its performance using five benchmark finite-horizon POMDPs, demonstrating its effectiveness in estimating upper bounds and improving PBVI performance. GP-UCB reduces computation time by 30% to 60% on smaller problems and up to 99.7% on larger ones, while achieving the same gaps as the pure sawtooth projection method.

Accelerating Point-Based Value Iteration via Active Sampling of Belief Points and Gaussian Process Regression

TL;DR

The paper tackles the computational bottleneck of upper bound estimation in finite-horizon POMDPs solved via PBVI. It introduces GP-UCB, which uses Gaussian Process Regression trained on a strategically selected subset of informative belief points to predict the upper bound convex hull, reducing reliance on costly sawtooth projections. The authors prove PAC-like consistency and demonstrate substantial speedups (30-60% on small problems, up to 99.7% on larger ones) while maintaining the same bound gaps as traditional sawtooth methods. Experiments on five benchmark POMDPs show improved scalability and competitive accuracy, highlighting GP-UCB as a practical tool for large-scale, finite-horizon POMDP planning with uncertain observations.

Abstract

Partially Observable Markov Decision Processes (POMDPs) are fundamental to decision-making under uncertainty. We introduce a novel scalable approach to accelerate upper bound estimation in Point-Based Value Iteration (PBVI) algorithms, the leading method to solve large-scale POMDPs. PBVI approximates the value function using a set of belief points rather than the entire continuous belief space and relies on lower and upper bounds for convergence. While lower bounds are straightforward to compute, PVBI requires repeated sawtooth projection operations to approximate the upper bound convex hull, significantly increasing the computational burden although many of these sawtooth projections become redundant as the belief set expands. To address this, we infer the upper bound using the upper confidence bound of a Gaussian Process Regression (GP-UCB) fitted over a subset of the most informative reachable belief points--the ones that exhibit linear independence in some high-dimensional Hilbert space. This approach reduces the number of sawtooth projections by 84.3% on average without compromising the solution quality. We further establish the theoretical consistency of the proposed GP-UCB estimate of the upper bound and show convergence to the true upper bound convex hull. We implement GP-UCB and test its performance using five benchmark finite-horizon POMDPs, demonstrating its effectiveness in estimating upper bounds and improving PBVI performance. GP-UCB reduces computation time by 30% to 60% on smaller problems and up to 99.7% on larger ones, while achieving the same gaps as the pure sawtooth projection method.

Paper Structure

This paper contains 15 sections, 2 theorems, 18 equations, 9 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Description
  3. PBVI Algorithm:
  4. Upper Bound Prediction using Gaussian Process Regression
  5. The Use of Gaussian Process Upper Confidence Bound
  6. Numerical Experiments
  7. Performance Results
  8. Conclusions
  9. Background on POMDP Problems and Solution Algorithms
  10. Point-Based Value Iteration
  11. Upper and Lower Bounds
  12. Sawtooth Projection to Obtain Upper Bounds
  13. Description of the tiger problem
  14. Further details on sawtooth projection method
  15. Full computational results

Key Result

Theorem 1

Denoting the convex hull upper bound for belief point $\bm{b}$ as ${h}(\bm{b})$, we have In other words, as more belief points are sampled, $\sigma(\bm{b})$ reduces and the mean prediction $\mu (\bm{b})$ approaches the true convex hull $h(\bm{b})$.

Figures (9)

  • Figure 1: Tree structure of a two-stage tiger problem kaelbling1998planning. Circles represent belief states observed right before taking an action in each stage.
  • Figure 2: Approximation of the upper bound using sawtooth projection across different time stages at iteration 2 in the tiger problem.
  • Figure 3: Upper bound estimation using GPR and sawtooth projection.
  • Figure 4: Upper bound estimation using Matérn and exponential kernels in the tiger problem iteration 1.
  • Figure 5: Estimation of upper bounds with the GPR model in the tiger problem.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Corollary 1
  • proof