A Pontryagin Maximum Principle on the Belief Space for Continuous-Time Optimal Control with Discrete Observations
Christian Bayer, Saifeddine Ben naamia, Erik von Schwerin, Raul Tempone
TL;DR
This work develops a Pontryagin maximum principle on the belief space $\mathcal{P}(\mathbb{R}^{d_x})$ for continuous-time stochastic control with discrete observations, formulating the controller's decisions as a pair $(\alpha_t,\beta_{t_n})$ and a finite memory $Z_{t_n}$. The theory yields a forward-backward system: a controlled filtering (Fokker–Planck) evolution for the belief between observations and Bayesian jumps at observation times, coupled with an adjoint that matches the gradient of the belief-value $V_t$ along the optimal path. The adjoint's jump includes a normalisation term analogous to Kushner–Stratonovich corrections, linking PMP to dynamic programming in the measure space. Numerically, a particle-based solver combines forward filtering with regression-based approximation of the adjoint/value, yielding locally optimal policies that actively manage information gathering under observation costs; results on linear and nonlinear problems, including LQG and non-LQG cases, demonstrate effectiveness and active sensing benefits.
Abstract
We study a continuous time stochastic optimal control problem under partial observations that are available only at discrete time instants. This hybrid setting, with continuous dynamics and intermittent noisy measurements, arises in applications ranging from robotic exploration and target tracking to epidemic control. We formulate the problem on the space of beliefs (information states), treating the controller's posterior distribution of the state as the state variable for decision making. On this belief space we derive a Pontryagin maximum principle that provides necessary conditions for optimality. The analysis carefully tracks both the continuous evolution of the state between observation times and the Bayesian jump updates of the belief at observation instants. A key insight is a relationship between the adjoint process in our maximum principle and the gradient of the value functional on the belief space, which links the optimality conditions to the dynamic programming approach on the space of probability measures. The resulting optimality system has a prediction and update structure that is closely related to the unnormalised Zakai equation and the normalised Kushner-Stratonovich equation in nonlinear filtering. Building on this analysis, we design a particle based numerical scheme to approximate the coupled forward (filter) and backward (adjoint) system. The scheme uses particle filtering to represent the evolving belief and regression techniques to approximate the adjoint, which yields a practical algorithm for computing near optimal controls under partial information. The effectiveness of the approach is illustrated on both linear and nonlinear examples and highlights in particular the benefits of actively controlling the observation process.