Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation of Singular-Stopping Control Driven by Hawkes Processes via Rescaled MDPs

Isabel Agostino, Thibaut Mastrolia

TL;DR

The paper addresses a singular-stopping stochastic control problem driven by Hawkes processes, motivated by cyber-risk contexts where attack clusters are self-exciting. It develops a continuous-time formulation characterized by a variational Hamilton-Jacobi-Bellman equation with a gradient constraint and then builds a discrete-time Markov decision process approximation whose value function converges to the continuous-time value under a suitable time rescaling. A rigorous convergence theory is established, including continuous-time interpolation, time-rescaling, and tightness, ensuring that discrete optimizers are asymptotically optimal for the continuous problem. The approach is demonstrated on a Hawkes-driven Ornstein–Uhlenbeck model for energy-investment under cyber threats, with numerical simulations validating the discretization and convergence results and illustrating optimal capital injections and stopping behavior.

Abstract

We investigate a singular-optimal stopping stochastic control problem driven by self-exciting dynamics governed by a Hawkes process. In the continuous-time setting, we show that the optimization problem reduces to solving a variational partial differential equation with gradient constraints. We then introduce its discrete-time counterpart, modeled as a Markov Decision Process. We prove that, under an appropriate rescaling procedure, the value function of the discrete-time problem converges to its continuous-time equivalent, implying that the discrete-time optimizers are asymptotically optimal for the continuous-time problem. Finally, we apply these results to an Ornstein-Uhlenbeck stochastic differential equation driven by a Hawkes process with singular control, motivated by optimal power plant investment under cyber threat and we illustrate the theoretical findings through numerical simulations.

Approximation of Singular-Stopping Control Driven by Hawkes Processes via Rescaled MDPs

TL;DR

The paper addresses a singular-stopping stochastic control problem driven by Hawkes processes, motivated by cyber-risk contexts where attack clusters are self-exciting. It develops a continuous-time formulation characterized by a variational Hamilton-Jacobi-Bellman equation with a gradient constraint and then builds a discrete-time Markov decision process approximation whose value function converges to the continuous-time value under a suitable time rescaling. A rigorous convergence theory is established, including continuous-time interpolation, time-rescaling, and tightness, ensuring that discrete optimizers are asymptotically optimal for the continuous problem. The approach is demonstrated on a Hawkes-driven Ornstein–Uhlenbeck model for energy-investment under cyber threats, with numerical simulations validating the discretization and convergence results and illustrating optimal capital injections and stopping behavior.

Abstract

We investigate a singular-optimal stopping stochastic control problem driven by self-exciting dynamics governed by a Hawkes process. In the continuous-time setting, we show that the optimization problem reduces to solving a variational partial differential equation with gradient constraints. We then introduce its discrete-time counterpart, modeled as a Markov Decision Process. We prove that, under an appropriate rescaling procedure, the value function of the discrete-time problem converges to its continuous-time equivalent, implying that the discrete-time optimizers are asymptotically optimal for the continuous-time problem. Finally, we apply these results to an Ornstein-Uhlenbeck stochastic differential equation driven by a Hawkes process with singular control, motivated by optimal power plant investment under cyber threat and we illustrate the theoretical findings through numerical simulations.
Paper Structure (30 sections, 22 theorems, 149 equations, 6 figures, 1 table)

This paper contains 30 sections, 22 theorems, 149 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Cyber Risk Management as a Motivating Framework
  3. State of the art and contribution
  4. The continuous time model
  5. Spaces and notations
  6. Hawkes process and exponential kernels
  7. Hawkes' formulation
  8. Brémaud-Massoulié's formulation
  9. Properties of the intensity process
  10. Assumption $\mathbf{(R)}$.
  11. Controlled SDE and singular control
  12. Stochastic optimal mixed stopping-singular control problem
  13. Interpretations.
  14. Restriction to a bounded domain
  15. Discrete Time Markov Chain Model
  16. ...and 15 more sections

Key Result

Lemma 2.2

For any $t\in [0,T]$ we have where $\bar{N}_t$ is a martingale and $\int_0^t\int_{\mathbb{R}_+}\int_{\mathbb{R}_+} \textbf{1}_{\theta\leq \lambda_s}dsd\theta \text{m}(dz)$ is the compensator of the Hawkes process. Moreover, for any random process $H(z)$ with $z\in \mathbb{R}_+$ such that the process $M^H$ defined by is an $(\mathbb P^\Pi,\mathbb F^\Pi)$-martingale.

Figures (6)

  • Figure 1: Convergence of $\mathcal{X}^h$ where $\mathcal{X}$ represents any generic process $X,H,B,M,H,R^+,R^-$
  • Figure 2: Exact OU diffusion vs. DTMC approximations.
  • Figure 3: Value function convergence in $x_0$ and $\lambda_0$
  • Figure 4: Convergence of the value function with fixed initial parameters for different $h$.
  • Figure 5: Top a path of $X,$bottom $V^{h=0.005}(x_0=0,\lambda_0=1)$ over time; red lines denote times of capital injection.
  • ...and 1 more figures

Theorems & Definitions (61)

  • Remark 2.1
  • Lemma 2.2: Doob-Meyer decomposition - Theorem 2.3.7. in applebaum2009levy
  • Lemma 2.3
  • proof
  • Lemma 2.4
  • proof
  • Lemma 2.5
  • proof
  • Definition 2.6: Admissible controls
  • Remark 2.7: Exponential Kernel dimension 1
  • ...and 51 more