Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A measure-valued HJB perspective on Bayesian optimal adaptive control

Alexander M. G. Cox, Sigrid Källblad, Chaorui Wang

TL;DR

This work develops a rigorous infinite-dimensional stochastic control framework for Bayesian adaptive control with an unobservable static signal that nonlinearly affects the observation drift. By treating the posterior as a measure-valued process and applying viscosity theory to a measure-space HJB, the authors establish equivalence between weak and approximate formulations and identify a unique continuous viscosity solution as the optimal value. A key contribution is the Barles-Souganides-style approximation in the infinite-dimensional setting, yielding both a convergence result and a practical, arbitrarily-close-to-optimal piecewise-constant control. The results also include a stability analysis for measure-valued SDEs, providing robustness of the posterior dynamics with respect to initial conditions. The framework advances understanding of information-control trade-offs in partially observed, nonlinear settings and offers a solid pathway for numerical construction of near-optimal policies.

Abstract

We consider a Bayesian adaptive optimal stochastic control problem where a hidden static signal has a non-separable influence on the drift of a noisy observation. Being allowed to control the specific form of this dependence, we aim at optimising a cost functional depending on the posterior distribution of the hidden signal. Our setup is in sharp contrast to existing work: we include costs that depend on the full posterior distribution in a form that admits a large class of non-linear relationships. Expressing the dynamics of this posterior distribution in the observation filtration, we embed our problem into a genuinely infinite-dimensional stochastic control problem using measure-valued martingales. We address this problem by use of viscosity theory and approximation arguments. Specifically, we show equivalence to a corresponding weak formulation, characterise the optimal value of the problem in terms of the unique continuous viscosity solution of an associated HJB equation, and construct a piecewise constant and arbitrarily-close-to-optimal control to our main problem of study. As a byproduct of our analysis, we also provide a novel stability result for a class of measure-valued SDEs which we believe is of independent interest.

A measure-valued HJB perspective on Bayesian optimal adaptive control

TL;DR

This work develops a rigorous infinite-dimensional stochastic control framework for Bayesian adaptive control with an unobservable static signal that nonlinearly affects the observation drift. By treating the posterior as a measure-valued process and applying viscosity theory to a measure-space HJB, the authors establish equivalence between weak and approximate formulations and identify a unique continuous viscosity solution as the optimal value. A key contribution is the Barles-Souganides-style approximation in the infinite-dimensional setting, yielding both a convergence result and a practical, arbitrarily-close-to-optimal piecewise-constant control. The results also include a stability analysis for measure-valued SDEs, providing robustness of the posterior dynamics with respect to initial conditions. The framework advances understanding of information-control trade-offs in partially observed, nonlinear settings and offers a solid pathway for numerical construction of near-optimal policies.

Abstract

We consider a Bayesian adaptive optimal stochastic control problem where a hidden static signal has a non-separable influence on the drift of a noisy observation. Being allowed to control the specific form of this dependence, we aim at optimising a cost functional depending on the posterior distribution of the hidden signal. Our setup is in sharp contrast to existing work: we include costs that depend on the full posterior distribution in a form that admits a large class of non-linear relationships. Expressing the dynamics of this posterior distribution in the observation filtration, we embed our problem into a genuinely infinite-dimensional stochastic control problem using measure-valued martingales. We address this problem by use of viscosity theory and approximation arguments. Specifically, we show equivalence to a corresponding weak formulation, characterise the optimal value of the problem in terms of the unique continuous viscosity solution of an associated HJB equation, and construct a piecewise constant and arbitrarily-close-to-optimal control to our main problem of study. As a byproduct of our analysis, we also provide a novel stability result for a class of measure-valued SDEs which we believe is of independent interest.
Paper Structure (16 sections, 16 theorems, 129 equations)

This paper contains 16 sections, 16 theorems, 129 equations.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Controls and objective
  4. MVMs and SDEs
  5. Weak and approximate formulation and main result
  6. Weak problem formulation
  7. Approximate problem formulation
  8. Associated HJB equation
  9. Our main result
  10. Stability properties of controlled MVMs
  11. Viscosity theory and proof of main result
  12. DPP for piecewise constant controls
  13. Subsolution property of limiting problem
  14. Proof of main result
  15. Proof of the DPP
  16. ...and 1 more sections

Key Result

Theorem 3.3

It holds that Moreover, on $\mathcal{P}$, and this function is the unique continuous viscosity solution of eq:hjb. Finally, for any $\varepsilon>0$, we can find $n\in\mathbb{N}$ and a function $\hat{u}^n:\mathcal{P}\to\mathcal{U}$, such that $u^*\in\mathcal{A}$ given in feedback form by satisfies

Theorems & Definitions (35)

  • Example 2.2
  • Remark 2.3: A finite-dimensional reduction
  • Remark 2.4
  • Definition 3.1: Weak controls
  • Remark 3.2
  • Theorem 3.3
  • Remark 3.4
  • Lemma 4.1
  • proof
  • Proposition 4.2
  • ...and 25 more