A Finite-State Controller Based Offline Solver for Deterministic POMDPs

TL;DR

DetMCVI tackles offline planning for DetPOMDPs by constructing policies as finite-state controllers using a Monte Carlo value iteration framework. The method introduces targeted backups, belief management, and a belief-tree search tailored to deterministic dynamics, yielding compact FSCs with strong goal achievement. Empirical results on synthetic DetPOMDP domains and a real-world forest navigation task show high success rates and much smaller policy sizes than baselines, with competitive planning times under budget. This approach enables scalable, resource-efficient offline planning for robotics in uncertain, topologically complex environments.

Abstract

Deterministic partially observable Markov decision processes (DetPOMDPs) often arise in planning problems where the agent is uncertain about its environmental state but can act and observe deterministically. In this paper, we propose DetMCVI, an adaptation of the Monte Carlo Value Iteration (MCVI) algorithm for DetPOMDPs, which builds policies in the form of finite-state controllers (FSCs). DetMCVI solves large problems with a high success rate, outperforming existing baselines for DetPOMDPs. We also verify the performance of the algorithm in a real-world mobile robot forest mapping scenario.

Figures (6)

• Figure 1: A topological map used for navigation in a forest where possibly obscured terrain leads to uncertain traversability.
• Figure 2: A partial belief tree and FSC built during MCVI iteration.
• Figure 3: Selected problem instances from each domain
• Figure 4: Success rate (left) and policy size (right) for different algorithms as evaluated on a CTP problem with $n=25$.
• Figure 5: Success rate of policies generated with downsampled initial beliefs. Left: CTP ${n=20}$, ${|\text{Supp}(b_0)| = 2048}$. Right: Wumpus ${n=3}$, ${|\text{Supp}(b_0)| \geq 10850}$.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Definition 3
• Example 1
• Definition 4
• Definition 5