Observation Adaptation via Annealed Importance Resampling for Partially Observable Markov Decision Processes

TL;DR

This work tackles online POMDP planning under state uncertainty, where standard particle filters suffer from degeneracy with highly informative observations. It proposes AIROAS, an online solver that integrates Annealed Importance Resampling with bridge distributions to gradually steer particles toward the optimal posterior $p(s_t|o_t,s_{t-1},a)$, reducing variance and improving belief accuracy. The method combines a Monte Carlo tree search framework with belief nodes and action nodes, upper/lower bound estimates, and an excess-uncertainty criterion to guide exploration, along with a Metropolis-Hastings mutation step to enrich particle diversity. Empirical evaluations across diverse domains show AIROAS outperforms state-of-the-art baselines, with ablations confirming that the AIR mechanism scales benefits with more particles and appropriate tempering, signaling significant practical impact for complex POMDPs in robotics and planning under uncertainty.

Abstract

Partially observable Markov decision processes (POMDPs) are a general mathematical model for sequential decision-making in stochastic environments under state uncertainty. POMDPs are often solved \textit{online}, which enables the algorithm to adapt to new information in real time. Online solvers typically use bootstrap particle filters based on importance resampling for updating the belief distribution. Since directly sampling from the ideal state distribution given the latest observation and previous state is infeasible, particle filters approximate the posterior belief distribution by propagating states and adjusting weights through prediction and resampling steps. However, in practice, the importance resampling technique often leads to particle degeneracy and sample impoverishment when the state transition model poorly aligns with the posterior belief distribution, especially when the received observation is highly informative. We propose an approach that constructs a sequence of bridge distributions between the state-transition and optimal distributions through iterative Monte Carlo steps, better accommodating noisy observations in online POMDP solvers. Our algorithm demonstrates significantly superior performance compared to state-of-the-art methods when evaluated across multiple challenging POMDP domains.

Figures (2)

• Figure 1: Process of AIROAS Tree Search. AIROAS constructs a search tree that alternates between belief nodes and action nodes. AIR represents Annealed Importance Resampling.
• Figure 2: Performance comparison between AIROAS variants with and without Annealed Importance Resampling (AIR) on the Light Dark domain ($\alpha = 1.0$). The performance gap widens in favor of AIR as the number of particles increases.

Theorems & Definitions (1)

• proof