Sequential Monte Carlo for Policy Optimization in Continuous POMDPs

TL;DR

The paper tackles optimal decision-making under partial observability in continuous POMDPs by casting policy optimization as probabilistic inference in a belief-space Feynman–Kac model. It develops a nested Sequential Monte Carlo framework (P3O) that samples from the optimal trajectory distribution and yields a policy gradient via Fisher's identity, enabling deliberate information gathering rather than relying on suboptimal approximations. Empirical results show improved performance on challenging tasks requiring exploration (e.g., light-dark, triangulation) and robustness across baselines that use QMDP approximations. The work advances the state of offline, belief-space policy optimization for continuous POMDPs and highlights practical considerations such as variance control, particle counts, and η-tuning. It lays a foundation for principled, information-seeking control in partially observable environments with potential extensions to higher dimensions and learning-based state estimators.

Abstract

Optimal decision-making under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploitation). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explicitly addresses this challenge. Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model that inherently captures the value of information gathering by anticipating future observations, without requiring suboptimal approximations or handcrafted heuristics. To optimize policies under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.

Figures (5)

• Figure 1: Benchmarking P3O on a linear-Gaussian quadratic problem. Left: LQG provides a tractable optimal baseline and P3O converges to a similar performance under general assumptions. Right: the influence of the temperature $\eta$ on the variance of the gradient, higher $\eta$ leads to smaller variance.
• Figure 2: Experiment results on various benchmarks. We report the average return using $1024$ trajectory rollouts and plot the mean and standard error over $10$ training seeds. P3O and DVRL consistently outperform the QMDP-based solvers, SLAC and DualSMC, which especially struggle in the light-dark and triangulation tasks that require deliberate exploration.
• Figure 3: Example trajectories from the light-dark (left) and triangulation (middle and right) tasks. In the light-dark environment, the agent learns to steer towards low-uncertainty regions to localize itself, then moves to the target. We visualize the agent's belief at every time step. In the triangulation task, the agent executes a specialized maneuver to estimate its position using only heading measurements. We plot the largest eigenvalue of the agent's belief covariance over time. The red dashed lines represent the shortest path the agent could follow if no information gathering was required.
• Figure 4: Influence of the number of particles in the Feynman–Kac filter $(N)$ and belief filter $(M)$. Increasing $N$ and $M$ improves the approximation of the target distribution $\Psi_{T}$ and the policy gradient, resulting in better learning performance. Plots depict the mean and standard error over $10$ seeds.
• Figure 5: The effect of the temperature $\eta$ on policy learning. Lower values of $\eta$ lead to high-variance policy gradients and slower convergence.

Theorems & Definitions (5)

• Proposition 1: Belief-Space Objective
• Proposition 2: POMDP Policy Gradient
• Remark 1
• proof
• proof