Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning Design for Quickest Change Detection

Austin Cooper, Sean Meyn

TL;DR

This paper develops reinforcement learning designs for quickest change detection (QCD) under partial statistical knowledge by leveraging surrogate information states derived from observations. It frames Bayesian QCD as a POMDP and analyzes two RL approaches—actor-critic SGD and Q-learning with a projected Bellman equation—alongside asymptotic results for threshold-based detectors like CUSUM. The work provides theoretical guarantees (unbiased gradients, stability under optimistic training) and extensive numerical evidence showing that Q-learning can yield near-optimal threshold policies, with performance within a few percent of the optimal benchmark in several scenarios. The findings offer a practical roadmap for designing RL-based QCD algorithms in general settings, balancing detection delays and false alarms while coping with mismatched or incomplete post-change statistics.

Abstract

The field of quickest change detection (QCD) concerns design and analysis of algorithms to estimate in real time the time at which an important event takes place, and identify properties of the post-change behavior. It is shown in this paper that approaches based on reinforcement learning (RL) can be adapted based on any "surrogate information state" that is adapted to the observations. Hence we are left to choose both the surrogate information state process and the algorithm. For the former, it is argued that there are many choices available, based on a rich theory of asymptotic statistics for QCD. Two approaches to RL design are considered: (i) Stochastic gradient descent based on an actor-critic formulation. Theory is largely complete for this approach: the algorithm is unbiased, and will converge to a local minimum. However, it is shown that variance of stochastic gradients can be very large, necessitating the need for commensurately long run times; (ii) Q-learning algorithms based on a version of the projected Bellman equation. It is shown that the algorithm is stable, in the sense of bounded sample paths, and that a solution to the projected Bellman equation exists under mild conditions. Numerical experiments illustrate these findings, and provide a roadmap for algorithm design in more general settings.

Reinforcement Learning Design for Quickest Change Detection

TL;DR

This paper develops reinforcement learning designs for quickest change detection (QCD) under partial statistical knowledge by leveraging surrogate information states derived from observations. It frames Bayesian QCD as a POMDP and analyzes two RL approaches—actor-critic SGD and Q-learning with a projected Bellman equation—alongside asymptotic results for threshold-based detectors like CUSUM. The work provides theoretical guarantees (unbiased gradients, stability under optimistic training) and extensive numerical evidence showing that Q-learning can yield near-optimal threshold policies, with performance within a few percent of the optimal benchmark in several scenarios. The findings offer a practical roadmap for designing RL-based QCD algorithms in general settings, balancing detection delays and false alarms while coping with mismatched or incomplete post-change statistics.

Abstract

The field of quickest change detection (QCD) concerns design and analysis of algorithms to estimate in real time the time at which an important event takes place, and identify properties of the post-change behavior. It is shown in this paper that approaches based on reinforcement learning (RL) can be adapted based on any "surrogate information state" that is adapted to the observations. Hence we are left to choose both the surrogate information state process and the algorithm. For the former, it is argued that there are many choices available, based on a rich theory of asymptotic statistics for QCD. Two approaches to RL design are considered: (i) Stochastic gradient descent based on an actor-critic formulation. Theory is largely complete for this approach: the algorithm is unbiased, and will converge to a local minimum. However, it is shown that variance of stochastic gradients can be very large, necessitating the need for commensurately long run times; (ii) Q-learning algorithms based on a version of the projected Bellman equation. It is shown that the algorithm is stable, in the sense of bounded sample paths, and that a solution to the projected Bellman equation exists under mild conditions. Numerical experiments illustrate these findings, and provide a roadmap for algorithm design in more general settings.
Paper Structure (10 sections, 4 theorems, 33 equations, 9 figures)

This paper contains 10 sections, 4 theorems, 33 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Bayesian QCD
  3. POMDP model
  4. Asymptotic statistics
  5. Reinforcement Learning and QCD
  6. Numerical Results with Q-learning
  7. Cost approximation
  8. Q-learning
  9. Conclusions
  10. Appendix

Key Result

Lemma 2.1

[lemma]t:POMDPhazard Consider the POMDP model with ${\sf X}$ finite, ${\sf P}\{ \uptau_{\text{\scriptsize\sf a}} = \infty \} = 0$, yet ${\sf P}\{ \uptau_{\text{\scriptsize\sf a}} > N \} >0$ for each $N>0$ and $\Phi_0 \in{\sf X}_1$. Then e:hazardAss holds for some $\varrho_{\text{\scriptsize\sf a}}

Figures (9)

  • Figure 1: Approximations of the optimal threshold and cost.
  • Figure 2: Statistics of the gradient estimate as a function of $\theta$, based on the Actor Critic method: (a) Empirical variance of the gradient estimates. (b) Gradient estimates using $N=10^4$ episodes. (c) Objective and its approximation obtained from integrating the gradient estimate.
  • Figure 3: Three LLRs
  • Figure 4: $\Lambda_0(\upupsilon)$ for ideal Gaussian alongside Laplace and Cauchy mismatched detectors
  • Figure 5: Parameter estimates using a decaying exploration schedule.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Lemma 2.1
  • Proposition 2.2
  • Proposition 3.1
  • Proposition 3.2