Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sub-optimality bounds for certainty equivalent policies in partially observed systems

Berk Bozkurt, Aditya Mahajan, Ashutosh Nayyar, Yi Ouyang

TL;DR

The work generalizes the certainty equivalence principle to partially observed stochastic systems by allowing any state estimator within CE policies and by embracing state abstractions. It leverages the approximate information state framework to derive explicit sub-optimality bounds that depend on the estimation error and Lipschitz-like smoothness of dynamics and costs. The results are instantiated through diverse examples (bounded/noise, degraded observations, quantization, learning, event-triggered sensing, and multi-particle systems) to demonstrate near-optimal performance when estimation is accurate. This provides a practical, scalable route to CE-based control in complex POMDPs with provable guarantees, bridging theory and application in robotics, finance, and beyond.

Abstract

In this paper, we present a generalization of the certainty equivalence principle of stochastic control. One interpretation of the classical certainty equivalence principle for linear systems with output feedback and quadratic costs is as follows: the optimal action at each time is obtained by evaluating the optimal state-feedback policy of the stochastic linear system at the minimum mean square error (MMSE) estimate of the state. Motivated by this interpretation, we consider certainty equivalent policies for general (non-linear) partially observed stochastic systems that allow for any state estimate rather than restricting to MMSE estimates. In such settings, the certainty equivalent policy is not optimal. For models where the cost and the dynamics are smooth in an appropriate sense, we derive upper bounds on the sub-optimality of certainty equivalent policies. We present several examples to illustrate the results.

Sub-optimality bounds for certainty equivalent policies in partially observed systems

TL;DR

The work generalizes the certainty equivalence principle to partially observed stochastic systems by allowing any state estimator within CE policies and by embracing state abstractions. It leverages the approximate information state framework to derive explicit sub-optimality bounds that depend on the estimation error and Lipschitz-like smoothness of dynamics and costs. The results are instantiated through diverse examples (bounded/noise, degraded observations, quantization, learning, event-triggered sensing, and multi-particle systems) to demonstrate near-optimal performance when estimation is accurate. This provides a practical, scalable route to CE-based control in complex POMDPs with provable guarantees, bridging theory and application in robotics, finance, and beyond.

Abstract

In this paper, we present a generalization of the certainty equivalence principle of stochastic control. One interpretation of the classical certainty equivalence principle for linear systems with output feedback and quadratic costs is as follows: the optimal action at each time is obtained by evaluating the optimal state-feedback policy of the stochastic linear system at the minimum mean square error (MMSE) estimate of the state. Motivated by this interpretation, we consider certainty equivalent policies for general (non-linear) partially observed stochastic systems that allow for any state estimate rather than restricting to MMSE estimates. In such settings, the certainty equivalent policy is not optimal. For models where the cost and the dynamics are smooth in an appropriate sense, we derive upper bounds on the sub-optimality of certainty equivalent policies. We present several examples to illustrate the results.
Paper Structure (38 sections, 8 theorems, 86 equations)

This paper contains 38 sections, 8 theorems, 86 equations.

Table of Contents

  1. Introduction
  2. System model and the main results
  3. Certainty equivalent policies
  4. Certainty equivalent policies using State Abstraction
  5. Illustrative examples
  6. Bounded observation noise
  7. System model
  8. Certainty equivalent policy
  9. Sub-optimality bound
  10. Intermittently degraded observation
  11. System model
  12. Certainty equivalent policy
  13. Sub-optimality bound
  14. Bounded observation noise with state quantization
  15. System model
  16. ...and 23 more sections

Key Result

Theorem 1

Define where $\eta_t$ is given by eq:closeness_assm. Then, under Assumptions ass:meas-selection, ass:lipschitz and ass:finite_eta, we have that the certainty equivalent policy $\mu^\mathcal{E}$ (defined in eq:cert_equiv) satisfies where and $\{V^{{\mathcal{M}}}_{t}\}_{t=1}^T$ are the optimal value functions for MDP ${\mathcal{M}}$.

Theorems & Definitions (19)

  • Definition 1: Measurable selection
  • Remark 1
  • Remark 2
  • Theorem 1
  • Remark 3
  • Theorem 2
  • Corollary 1
  • Proof 1
  • Lemma 1
  • Remark 4
  • ...and 9 more