Robust Recurrence of Discrete-Time Infinite-Horizon Stochastic Optimal Control with Discounted Cost

TL;DR

This work addresses stability for discrete-time stochastic systems controlled by stationary policies that minimize an infinite-horizon discounted cost $J_\gamma$, with discount factor $\gamma\in(0,1)$. The authors fuse Bellman equation methods with a Lyapunov-like construct that leverages a storage function $W$ and a measurement function $\sigma(x)$ to prove semi-global practical recurrence of the closed-loop dynamics, uniform in $\gamma$. The key contributions include stochastic cost-controllability and detectability assumptions that generalize deterministic results, a rigorous proof of semi-global practical recurrence, and robustness results under mild continuity and small perturbations. This provides stability guarantees for broad nonlinear stochastic systems under near-optimal policies and supports resilience to model mismatch in practical implementations.

Abstract

We analyze the stability of general nonlinear discrete-time stochastic systems controlled by optimal inputs that minimize an infinite-horizon discounted cost. Under a novel stochastic formulation of cost-controllability and detectability assumptions inspired by the related literature on deterministic systems, we prove that uniform semi-global practical recurrence holds for the closed-loop system, where the adjustable parameter is the discount factor. Under additional continuity assumptions, we further prove that this property is robust.