ESCORT: Efficient Stein-variational and Sliced Consistency-Optimized Temporal Belief Representation for POMDPs

TL;DR

ESCORT addresses the challenge of representing complex, high-dimensional, and multi-modal beliefs in POMDPs by extending SVGD with correlation-aware projections and temporal consistency. The method combines deterministic particle evolution with a model-based belief update, learning projection matrices that preserve inter-dimensional correlations and constrain belief evolution over time. Empirical results across diverse POMDP domains and synthetic benchmarks show substantial improvements in belief fidelity and downstream decision quality, with ablations highlighting the critical role of correlation-aware regularization and temporal consistency. The approach offers scalable, modular belief representations that integrate with existing POMDP solvers, advancing practical decision-making under uncertainty.

Abstract

In Partially Observable Markov Decision Processes (POMDPs), maintaining and updating belief distributions over possible underlying states provides a principled way to summarize action-observation history for effective decision-making under uncertainty. As environments grow more realistic, belief distributions develop complexity that standard mathematical models cannot accurately capture, creating a fundamental challenge in maintaining representational accuracy. Despite advances in deep learning and probabilistic modeling, existing POMDP belief approximation methods fail to accurately represent complex uncertainty structures such as high-dimensional, multi-modal belief distributions, resulting in estimation errors that lead to suboptimal agent behaviors. To address this challenge, we present ESCORT (Efficient Stein-variational and sliced Consistency-Optimized Representation for Temporal beliefs), a particle-based framework for capturing complex, multi-modal distributions in high-dimensional belief spaces. ESCORT extends SVGD with two key innovations: correlation-aware projections that model dependencies between state dimensions, and temporal consistency constraints that stabilize updates while preserving correlation structures. This approach retains SVGD's attractive-repulsive particle dynamics while enabling accurate modeling of intricate correlation patterns. Unlike particle filters prone to degeneracy or parametric methods with fixed representational capacity, ESCORT dynamically adapts to belief landscape complexity without resampling or restrictive distributional assumptions. We demonstrate ESCORT's effectiveness through extensive evaluations on both POMDP domains and synthetic multi-modal distributions of varying dimensionality, where it consistently outperforms state-of-the-art methods in terms of belief approximation accuracy and downstream decision quality.

Figures (10)

• Figure 1: Overview of the ESCORT framework. The diagram illustrates the iterative process of maintaining accurate belief representations in POMDPs through deterministic particle evolution. Purple dots represent particles, yellow regions show distribution modes, and colored arrows indicate different force types. In the Temporal Consistency component, green arrows represent permissible belief transitions while red crossed arrows indicate prevented unrealistic jumps.
• Figure 2: The figure illustrates how GSWD regularization prevents invalid belief jumps between consecutive timesteps. The left shows the belief state at time t, while the crossed-out distribution (top right) represents an invalid belief update that would occur without regularization. The bottom right shows the valid belief at t+1 after applying temporal consistency constraints.
• Figure 3: Light Dark 10D POMDP Environment. Four 2D projections of the 10D state space (5D position + 5D velocity) showing: (Top-left) Symmetric corridors creating perceptual aliasing with multi-modal belief particles (orange) splitting in dark regions; (Top-right) Z-shaped navigation path with sequential waypoints; (Bottom-left) Start-to-goal trajectory from dark region (red circle) to goal beacon (green star); (Bottom-right) Cross-dimensional correlations. Light intensity maps (blue-to-white) encode observation noise $\sigma^2(\mathbf{x}) = 0.5^2 \times (1 - L(\mathbf{x})) + 0.01^2$, with contour lines marking light boundaries. The environment challenges belief approximation through high dimensionality, multi-modality from symmetric patterns, and strong position-velocity correlations ($\rho = 0.8$).
• Figure 4: Kidnapped Robot Problem visualization showing domain. The robot icon (blue eyes with red antenna) indicates the true robot position and orientation. Various landmark types are distributed across the map: houses (red roofs), shops (red and white striped awnings), and warehouses (with forklift symbols), creating perceptually similar patterns that cause aliasing. The green dashed circle shows the sensor range (radius = 5), and green dashed lines indicate the 90° field of view. Orange dots represent belief particles from ESCORT, with particle density shown as a yellow-red heatmap overlay. The robot must localize itself despite ambiguous observations from these visually similar landmark configurations.
• Figure 5: Multiple Target Tracking Environment. An agent (drone) with limited field of view (cyan wedge) must navigate to the goal (star) while tracking multiple independently moving targets (cars with velocity arrows) across zones with varying observability: Clear View (full visibility), Limited Sensing (30% visibility), Confusion Zone (identity swaps possible), and Blind Spot (no visibility). The 20D continuous state space and partial observability create multi-modal belief distributions.

Theorems & Definitions (5)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5