Numerical method to solve impulse control problems for partially observed piecewise deterministic Markov processes

TL;DR

The paper tackles numerical solutions for impulse control of partially observed PDMPs with hidden jump times by formulating the problem as a POMDP and then converting it to a belief-state MDP. It introduces a two-stage discretization approach—first on the Euclidean state space and then on the belief space—to produce tractable dynamic programming recursions with explicit error bounds that depend on discretization diameters. The authors provide regularity assumptions ensuring the operators remain well-behaved and prove convergence results for the discretized value functions, accompanied by a practical candidate strategy. The methodology is illustrated through simulations in medical treatment optimization, with broader applicability to biology and dynamic reliability. The work offers a viable path for solving complex decision problems under partial observation in continuous-time, hybrid systems with jumps, and it provides open avenues for proving near-optimality of the constructed policy.

Abstract

Designing efficient and rigorous numerical methods for sequential decision-making under uncertainty is a difficult problem that arises in many applications frameworks. In this paper we focus on the numerical solution of a subclass of impulse control problem for piecewise deterministic Markov process (PDMP) when the jump times are hidden. We first state the problem as a partially observed Markov decision process (POMDP) on a continuous state space and with controlled transition kernels corresponding to some specific skeleton chains of the PDMP. Then we proceed to build a numerically tractable approximation of the POMDP by tailor-made discretizations of the state spaces. The main difficulty in evaluating the discretization error comes from the possible random jumps of the PDMP between consecutive epochs of the POMDP and requires special care. Finally we discuss the practical construction of discretization grids and illustrate our method on simulations.

Figures (8)

• Figure 1: Graphical sketch of the approximation procedure. The first column corresponds to the original PDMP / POMDP / fully observed MDP. The second column corresponds to the first discretization. The third column corresponds to the second discretization.
• Figure 2: State graph for the changes of mode in the medical example. Letters indicate under which treatments the jumps are possible. Black arrows indicate changes to remission phase, red arrows changes to standard relapse, and blue arrows changes to therapeutic escape.
• Figure 3: Example of a controlled (OS) simulated strategy. The top panel illustrates the (hidden) values of the true process, with modes indicated by color, and allocated treatment indicated by point shape. The middle panel illustrates the observed trajectory (noisy version of the euclidean variable from the panel above). The bottom panel illustrates the most likely element of $\Omega$ estimated by the filter of the process.
• Figure 4: State graph. Full line arrows indicate deterministic jumps at the boundary, while dashed arrows indicate stochastic jumps. Letters indicate under which treatments the jumps are possible.
• Figure 5: Example of controlled trajectory. Representation of coordinate $\zeta$ of a controlled process. The process is in mode $0$ when drawn in black, in mode $1$ when in green, and in mode $2$ when in red. The $T_i$ represent hidden natural jump times. Letters above indicate the changes of treatment that can only occur at dates in the time grid $\delta^{1:N}=\{\delta,2\delta,\ldots, N\delta\}$.

Theorems & Definitions (33)

• Proposition 3.8
• Theorem 3.9
• Theorem 3.10
• Theorem 3.11
• Lemma A.1
• proof
• Lemma A.2
• proof
• Lemma A.3
• proof