Open-loop POMDP Simplification and Safe Skipping of Replanning with Formal Performance Guarantees

Abstract

Partially Observable Markov Decision Processes (POMDPs) provide a principled mathematical framework for decision-making under uncertainty. However, the exact solution to POMDPs is computationally intractable. In this paper, we address the computational intractability by introducing a novel framework for adaptive open-loop simplification with formal performance guarantees. Our method adaptively interleaves open-loop and closed-loop planning via a topology-based belief tree, enabling a significant reduction in planning complexity. The key contribution lies in the derivation of efficiently computable bounds which provide formal guarantees and can be used to ensure that our simplification can identify the immediate optimal action of the original POMDP problem. Our framework therefore provides computationally tractable performance guarantees for macro-actions within POMDPs. Furthermore, we propose a novel framework for safely skipping replanning during execution, supported by theoretical guarantees on multi-step open-loop action sequences. To the best of our knowledge, this framework is the first to address skipping replanning with formal performance guarantees. Practical online solvers for our proposed simplification are developed, including a sampling-based solver and an anytime solver. Empirical results demonstrate substantial computational speedups while maintaining provable performance guarantees, advancing the tractability and efficiency of POMDP planning.

Figures (8)

• Figure 1: A hybrid belief tree demonstrating the computational advantage of open-loop planning. The left branch employs open-loop action sequences, while the right branch utilizes traditional closed-loop planning.
• Figure 2: Illustration of two cases for performance guarantees: (a) overlapping bounds necessitating topology refinement, and (b) non-overlapping bounds allowing for optimal action determination.
• Figure 3: The process of skipping replanning based on the allowed observation set $\bar{\mathcal{Z}}_1$. The agent executes the action $a^*_0$ and checks if the observation $z_{1}$ belongs to the set $\bar{\mathcal{Z}}_{1}$. If it does, the agent skips replanning; otherwise, it triggers replanning. The process to skip replanning with performance guarantee is shown in dotted lines, which can be conducted in parallel with the execution of action and observation, thus reducing the replanning overhead.
• Figure 4: Distribution of estimated bounds. The upper and lower bounds are computed using our proposed method with open-loop simplification, AT-SparsePFT. The yellow triangle denotes the $Q$-value estimated by the standard SparsePFT.
• Figure 5: Skipping replanning with guarantees: (a) Planning with the proposed adaptive open-loop simplification at $t=0$. It shows $ub(\tau, b_0,a_0)$ and $lb(\tau, b_0,a_0)$ for different $a_0$. $a_0^*$ is identified. (b) Bounding $Q^{*}(b_1,a_1)$ at $t=0$ for any posterior belief $b_1$ that corresponds to future observations in $\bar{\mathcal{Z}}_1$. The figure shows $\bar{ub}^{1}$ and $\bar{lb}^{1}$ defined in Theorem \ref{['theorem:bound-step-2-z-region']}. Here $a_1^*$ can be provably determined at $t=0$ for all realizations of the future observation $z_1$ in $\bar{\mathcal{Z}}_1$.

Theorems & Definitions (15)

• theorem 1
• proof
• theorem 2
• proof
• theorem 3: Convergence of AT-POMCP
• proof
• theorem 4
• proof
• proposition 1
• theorem 5