Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Explore with Belief: State Entropy Maximization in POMDPs

Riccardo Zamboni, Duilio Cirino, Marcello Restelli, Mirco Mutti

TL;DR

This work extends state entropy maximization to partially observable environments by maximizing the entropy of the true state distribution $H(d^\pi)$ using policies that rely only on partial observations. It introduces proxy objectives—Maximum Observation Entropy (MOE) and Maximum Believed Entropy (MBE)—and a Belief-Averaged (BA) policy class to enable tractable learning via policy gradients, even when the true state is not observable. To address the gap between proxies and the true objective, the authors add a belief-entropy regularization (Reg-MBE) and analyze the optimization landscape, including smoothness bounds and proxy gaps, under known and unknown POMDP settings. Empirical results on gridworld POMDPs show MOE is fragile to observation quality, while MBE and Reg-MBE provide robust improvements, with MSE achieving the best performance when the POMDP is fully known. Overall, the paper demonstrates a practical, deployable approach to exploration in partially observable domains and identifies hallucinatory risks and mitigation strategies for belief-based objectives.

Abstract

Recent works have studied *state entropy maximization* in reinforcement learning, in which the agent's objective is to learn a policy inducing high entropy over states visitation (Hazan et al., 2019). They typically assume full observability of the state of the system, so that the entropy of the observations is maximized. In practice, the agent may only get *partial* observations, e.g., a robot perceiving the state of a physical space through proximity sensors and cameras. A significant mismatch between the entropy over observations and true states of the system can arise in those settings. In this paper, we address the problem of entropy maximization over the *true states* with a decision policy conditioned on partial observations *only*. The latter is a generalization of POMDPs, which is intractable in general. We develop a memory and computationally efficient *policy gradient* method to address a first-order relaxation of the objective defined on *belief* states, providing various formal characterizations of approximation gaps, the optimization landscape, and the *hallucination* problem. This paper aims to generalize state entropy maximization to more realistic domains that meet the challenges of applications.

How to Explore with Belief: State Entropy Maximization in POMDPs

TL;DR

This work extends state entropy maximization to partially observable environments by maximizing the entropy of the true state distribution using policies that rely only on partial observations. It introduces proxy objectives—Maximum Observation Entropy (MOE) and Maximum Believed Entropy (MBE)—and a Belief-Averaged (BA) policy class to enable tractable learning via policy gradients, even when the true state is not observable. To address the gap between proxies and the true objective, the authors add a belief-entropy regularization (Reg-MBE) and analyze the optimization landscape, including smoothness bounds and proxy gaps, under known and unknown POMDP settings. Empirical results on gridworld POMDPs show MOE is fragile to observation quality, while MBE and Reg-MBE provide robust improvements, with MSE achieving the best performance when the POMDP is fully known. Overall, the paper demonstrates a practical, deployable approach to exploration in partially observable domains and identifies hallucinatory risks and mitigation strategies for belief-based objectives.

Abstract

Recent works have studied *state entropy maximization* in reinforcement learning, in which the agent's objective is to learn a policy inducing high entropy over states visitation (Hazan et al., 2019). They typically assume full observability of the state of the system, so that the entropy of the observations is maximized. In practice, the agent may only get *partial* observations, e.g., a robot perceiving the state of a physical space through proximity sensors and cameras. A significant mismatch between the entropy over observations and true states of the system can arise in those settings. In this paper, we address the problem of entropy maximization over the *true states* with a decision policy conditioned on partial observations *only*. The latter is a generalization of POMDPs, which is intractable in general. We develop a memory and computationally efficient *policy gradient* method to address a first-order relaxation of the objective defined on *belief* states, providing various formal characterizations of approximation gaps, the optimization landscape, and the *hallucination* problem. This paper aims to generalize state entropy maximization to more realistic domains that meet the challenges of applications.
Paper Structure (31 sections, 4 theorems, 27 equations, 6 figures, 1 table, 1 algorithm)

This paper contains 31 sections, 4 theorems, 27 equations, 6 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminaries
  4. Partially Observable MDPs
  5. State Entropy Maximization
  6. Problem Formulation
  7. MSE with a POMDP Simulator
  8. Smoothness of the Optimization Landscape.
  9. MSE without a POMDP Simulator
  10. Numerical Experiments
  11. Experimental Set-Up
  12. MSE in POMDP with the Proxy Objectives
  13. The Impact of Belief Approximation
  14. Related Work
  15. Conclusion
  16. ...and 16 more sections

Key Result

Theorem 4.2

For a policy $\pi_\theta \in \Pi_\mathcal{I}$ parametrized by $\theta \in \Theta \subseteq \mathbb{R}^{IA}$, we have where $\nabla_\theta \log \pi_\theta (\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t | i_t)$, $i \in \{\mathcal{S},\mathcal{O}\}$.

Figures (6)

  • Figure 1: MBE Proxy gaps: for different hallucination probabilities $\bar{p}_\mathcal{S}$ and a fixed trajectory $\tau_s$, the y-axis represents the possible MSE values contained between the upper bound and lower bound as $\mathcal{\tilde{J}}^\mathcal{S}(\pi|\tau_s)$ varies between $0$ and the maximum value $\log(S)$ (the corresponding MBE values are plotted over the diagonal to allow a comparisons).
  • Figure 2: True state entropy (or regularization term) obtained by Algorithm \ref{['alg:pg_pomdp']} specialized for the feedbacks MSE, MOE, MBE, MBE with belief regularization (Reg-MBE). For each plot, we report a tuple (environment, transition noise, observation variance) where the latter is not available (n.a.) when observations are deterministic. For each curve, we report the average and 95% c.i. over 16 runs.
  • Figure 3: True state entropy obtained by Algorithm \ref{['alg:pg_pomdp']} with MBE, MBE with belief regularization (MBE with Reg) feedbacks under different levels of approximation noise $s^2$. For each plot, we report a tuple (environment, transition noise, observation variance) where the latter is not available (n.a.) when observations are deterministic. For each curve, we report the average and 95% c.i. over 16 runs.
  • Figure 4: Environments Visualization.
  • Figure 5: True state entropy obtained by Algorithm $1$ specialized for the feedbacks MSE, MOE, MBE, MBE with belief regularization (Reg-MBE) over different policy classes with direct parametrization: Markovian over observation (O), Belief Averaged (BA), Markovian over hallucinated states (S). For each plot, we report a tuple (environment, transition noise, observation variance) where the latter is not available (n.a.) when observations are deterministic. For each curve, we report the average and 95% c.i. over 16 runs. BA confirms to be the policy class with generally higher performance in all the considered instances.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Theorem 4.2: Entropy Policy Gradient in POMDPs
  • Theorem 4.3: Local Lipschitz Constants
  • Theorem 5.2
  • Definition 5.3
  • Theorem 5.4: Proxy Gaps