Robust estimation of a Markov chain transition matrix from multiple sample paths

TL;DR

We address the problem of estimating a Markov chain transition matrix $P$ and stationary distribution $\\pi$ from $M$ independent sample paths of length $T$ each, allowing heterogeneous per-path dynamics $P_m$, initial laws $\\mu_m$, and stationary laws $\\pi_m$ with pseudo-spectral gaps $\\gamma_m$. The authors develop a general framework based on ensemble-time averages and derive sharp nonasymptotic Bernstein-type concentration bounds for the empirical transition matrix $\\hat{P}$ and stationary distribution $\\hat{\\pi}$, with explicit dependence on heterogeneity measures $\\Delta_1$, $\\Delta_\\infty$, and $\\eta$, and an effective mixing time $T'$; they also address robust estimation under fully corrupted rows and provide large-scale consistency results. The main contributions include (i) extending concentration results from single-path Markov chains to ensembles of heterogeneous chains, (ii) giving robust bounds that tolerate a fraction of corrupted trajectories, and (iii) establishing consistency in high-dimensional regimes where the number of states, chains, and path lengths may grow. The results yield practical guidance for inference in heterogeneous Markov models encountered in cohort studies, distributed systems, and temporal networks, and they are complemented by numerical experiments that illustrate the trade-offs between number of chains $M$ and trajectory length $T$ under various noise and mixing conditions.

Abstract

Markov chains are fundamental models for stochastic dynamics, with applications in a wide range of areas such as population dynamics, queueing systems, reinforcement learning, and Monte Carlo methods. Estimating the transition matrix and stationary distribution from observed sample paths is a core statistical challenge, particularly when multiple independent trajectories are available. While classical theory typically assumes identical chains with known stationary distributions, real-world data often arise from heterogeneous chains whose transition kernels and stationary measures might differ from a common target. We analyse empirical estimators for such parallel Markov processes and establish sharp concentration inequalities that generalise Bernstein-type bounds from standard time averages to ensemble-time averages. Our results provide nonasymptotic error bounds and consistency guarantees in high-dimensional regimes, accommodating sparse or weakly mixing chains, model mismatch, nonstationary initialisations, and partially corrupted data. These findings offer rigorous foundations for statistical inference in heterogeneous Markov chain settings common in modern computational applications.

Figures (2)

• Figure 1: Transition matrix estimation error for varying chain count $M$ (left) with respect to fixed total sample count $MT=10^4$ and state count ${\lvert\Omega\rvert}=10$; and varying state count ${\lvert\Omega\rvert}$ (right) with $M=50$ and $T=50$. Results are averaged over 50 simulations, and error bars display standard deviations for two noise levels ($\epsilon=0$ and $\epsilon=0.05$) with $\gamma=0.1$.
• Figure 2: Stationary distribution (left) and transition matrix (right) estimation error for varying transition rate parameter $\gamma$, with $|\Omega| = 10$, $M=200$, $T=200$. Results are averaged over 50 simulations, and error bars display standard deviations for two noise levels ($\epsilon=0$ and $\epsilon=0.03$).

Theorems & Definitions (29)

• Theorem 3.1
• Theorem 3.2
• Example 3.3: Clean data
• Example 3.4: Mixture of correct and misspecified chains
• Theorem 3.5
• Remark 3.6
• Corollary 3.7: Transition matrix consistency
• Corollary 3.8: Stationary distribution consistency
• Example 3.9: Random walk on the complete graph
• Remark 6.1