Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Deep Generative Models for Belief State Planning

Antoine Bigeard, Anthony Corso, Mykel Kochenderfer

TL;DR

This work tackles belief-state representation in high-dimensional POMDPs by introducing conditional deep generative models (cDGMs) that generate posterior samples conditioned on action-observation histories. Trained on random rollout data, cDGMs—specifically GANs and DDPMs—learn to sample from the posterior without explicit likelihoods, enabling scalable planning. In a mineral exploration POMDP with a large continuous state space, DDPM-based cDGMs outperform particle filters and GANs in both belief quality and planning performance when paired with a VOI-driven planner. The study emphasizes task-specific metrics for belief evaluation and demonstrates practical benefits for planning under uncertainty in complex, real-world-like domains.

Abstract

Partially observable Markov decision processes (POMDPs) are used to model a wide range of applications, including robotics, autonomous vehicles, and subsurface problems. However, accurately representing the belief is difficult for POMDPs with high-dimensional states. In this paper, we propose a novel approach that uses conditional deep generative models (cDGMs) to represent the belief. Unlike traditional belief representations, cDGMs are well-suited for high-dimensional states and large numbers of observations, and they can generate an arbitrary number of samples from the posterior belief. We train the cDGMs on data produced by random rollout trajectories and show their effectiveness in solving a mineral exploration POMDP with a large and continuous state space. The cDGMs outperform particle filter baselines in both task-agnostic measures of belief accuracy as well as in planning performance.

Conditional Deep Generative Models for Belief State Planning

TL;DR

This work tackles belief-state representation in high-dimensional POMDPs by introducing conditional deep generative models (cDGMs) that generate posterior samples conditioned on action-observation histories. Trained on random rollout data, cDGMs—specifically GANs and DDPMs—learn to sample from the posterior without explicit likelihoods, enabling scalable planning. In a mineral exploration POMDP with a large continuous state space, DDPM-based cDGMs outperform particle filters and GANs in both belief quality and planning performance when paired with a VOI-driven planner. The study emphasizes task-specific metrics for belief evaluation and demonstrates practical benefits for planning under uncertainty in complex, real-world-like domains.

Abstract

Partially observable Markov decision processes (POMDPs) are used to model a wide range of applications, including robotics, autonomous vehicles, and subsurface problems. However, accurately representing the belief is difficult for POMDPs with high-dimensional states. In this paper, we propose a novel approach that uses conditional deep generative models (cDGMs) to represent the belief. Unlike traditional belief representations, cDGMs are well-suited for high-dimensional states and large numbers of observations, and they can generate an arbitrary number of samples from the posterior belief. We train the cDGMs on data produced by random rollout trajectories and show their effectiveness in solving a mineral exploration POMDP with a large and continuous state space. The cDGMs outperform particle filter baselines in both task-agnostic measures of belief accuracy as well as in planning performance.
Paper Structure (19 sections, 10 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 19 sections, 10 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. POMDPs
  4. Particle Approximations for Bayesian Inference
  5. Deep Generative Models
  6. Generative Adversarial Networks (GANs)
  7. Denoising Diffusion Probabilistic Models (DDPMs)
  8. Mineral Exploration POMDP
  9. Conditional Deep Generative Models
  10. Conditioning the models
  11. Belief Evaluation Metrics
  12. Task-agnostic metrics
  13. Task-specific metrics
  14. Planning with cDGMs
  15. Experiments
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Summary of proposed approach
  • Figure 2: GAN Conditioning
  • Figure 3: DDPM Conditioning on step $t$ of denoising.
  • Figure 4: Samples generated from each belief representation. The top row consists of different ground-truth samples, and each other row contains the closest generated sample (out of 500.0) for each belief representation. The white circles which drilling locations and were used for conditioning the sample. The color inside the white circle shows the ground truth observation.
  • Figure 5: Belief metrics vs. number of actions/observations.