Asymptotically Optimal Sequential Testing with Markovian Data

TL;DR

The paper introduces a rigorously developed framework for one-sided sequential hypothesis testing with Markovian data under composite null and composite alternative hypotheses. It establishes a non-asymptotic, instance-dependent lower bound on the stopping time that scales with $\log(1/α)$ divided by the stationary-weighted KL divergence $D_{\mathcal{M}}^{\inf}(Q, 𝒫)$, and provides an asymptotically optimal test that attains this bound to first order as $α\to 0$. Central technical tools include a Pinsker-type inequality for Markov divergences via the Poisson equation and uniform control of Poisson solutions, enabling sharp finite-sample guarantees. The framework is demonstrated through applications to MCMC misspecification detection and to testing linearity in MDPs, illustrating both theoretical sharpness and practical applicability. The work thus yields a sharp, general characterization of optimal sequential testing procedures under Markovian dependence and opens avenues for extension to broader dependence models and non-convex hypothesis classes.

Abstract

We study one-sided and $α$-correct sequential hypothesis testing for data generated by an ergodic Markov chain. The null hypothesis is that the unknown transition matrix belongs to a prescribed set $P$ of stochastic matrices, and the alternative corresponds to a disjoint set $Q$. We establish a tight non-asymptotic instance-dependent lower bound on the expected stopping time of any valid sequential test under the alternative. Our novel analysis improves the existing lower bounds, which are either asymptotic or provably sub-optimal in this setting. Our lower bound incorporates both the stationary distribution and the transition structure induced by the unknown Markov chain. We further propose an optimal test whose expected stopping time matches this lower bound asymptotically as $α\to 0$. We illustrate the usefulness of our framework through applications to sequential detection of model misspecification in Markov Chain Monte Carlo and to testing structural properties, such as the linearity of transition dynamics, in Markov decision processes. Our findings yield a sharp and general characterization of optimal sequential testing procedures under Markovian dependence.

Figures (5)

• Figure 1: Test statistic trajectories over time for $Q_{\text{bad}}$ (red) and $Q_{\text{good}}$ (green), aggregated over $100$ runs. Solid curves and shaded regions show the mean $\pm 3\sigma$; dotted colored curves denote the boundary. The dotted black line shows the theoretical slope $D_{\mathcal{M}}^{\inf}(Q_{\text{bad}},\mathcal{P}_\pi)$, and the vertical line marks the mean stopping time for $Q_{\text{bad}}$.
• Figure 2: Mean statistic trajectory aggregated over $20$ runs. Shaded regions indicate $\pm 3$ standard deviations.
• Figure 5: Expected Stopping Time as a function of $\log\left(\frac{1}{\alpha}\right)$ for the experimental setup described in Appendix \ref{['experiments_on_parametric']}.
• Figure 6: Expected stopping time as a function of $\theta \in [-0.8,-0.4]$ for the experimental setup described in Section \ref{['baseline comparisions']}.
• Figure 7: Expected Stopping Time as a function of $\log\left(\frac{1}{\alpha}\right)$ for a fixed $\theta=-0.6$ in the experimental setup described in \ref{['baseline comparisions']}.

Theorems & Definitions (69)

• Definition 2.2: Pseudo-spectral gap, paulin2018concentrationinequalitiesmarkovchains
• Proposition 3.1
• Definition 3.2: Stationary-weighted KL divergence
• Theorem 3.3: Lower bound
• Remark 3.4: Recovery of i.i.d. bounds
• Theorem 4.1
• Remark 4.2
• Proposition 4.3
• Theorem 4.4
• Corollary 5.1