Stochastic Approximation in a Markovian Framework Revisited: Lipschitz Continuity of the Poisson Equation
Algo Carè, Balázs Csanád Csáji, Balázs Gerencsér, László Gerencsér, Miklós Rásonyi
TL;DR
This paper addresses the challenge of establishing existence, uniqueness, and Lipschitz-continuity of the parameter-dependent Poisson equation in a Markovian SA framework. By leveraging Hairer–Mattingly contractions in a weighted total variation space and introducing a simple Lipschitz condition on the one-step kernels, the authors derive concrete bounds ensuring the Poisson solution depends Lipschitz-continuously on the parameter. They extend these results under relaxed drift conditions, and demonstrate applicability through an open-loop controlled queueing system, illustrating the method's practicality for RL, HMM analysis, and cyber-physical contexts. Overall, the work provides a streamlined, verifiable set of assumptions to guarantee key regularity properties essential for ODE-based SA analysis in Markovian environments, with potential broad impact on online learning and stochastic control.
Abstract
In this paper we revisit a fundamental technical issue within the theory of stochastic approximation (SA) in a Markovian framework, first proposed in the book by Djereveckii and Fradkov (1981), and further developed in much detail in the book by Benveniste, M{é}tivier, and Priouret (1990). This theory is instrumental in many application areas such as the statistical analysis of Hidden Markov Models arising in telecommunication, quantized linear stochastic systems, and more recently in active learning and reinforcement learning. The problem at hand is the verification of the existence, uniqueness and Lipschitz-continuity of the solution of a parameter-dependent Poisson equation, in an appropriate weighted sup-norm, associated with a collection of Markov chains on general state spaces. Verification of the above facts is vital in the analysis of SA processes presented in (Benveniste et al., 1990) via the ODE (ordinary differential equations) method, requiring substantial technical effort. The motivation and focus of the paper is to address this technical issue, by presenting a simple set of conditions, under which the above properties of the Poisson equation at hand can be conveniently established. The starting point of our work is an intricate result of Hairer and Mattingly (2011) proving that by tilting standard conditions of mainstream stability theory for Markov chains, the transition kernels prove to be contractions in the space of differences of probability measures in a suitable metric. To demonstrate the applicability of our results, the proposed conditions are verified for a class of queuing system with open-loop control.