Data-Driven Non-Parametric Model Learning and Adaptive Control of MDPs with Borel spaces: Identifiability and Near Optimal Design
Omar Mrani-Zentar, Serdar Yüksel
TL;DR
The paper tackles average-cost stochastic control with unknown, nonparametric transition kernels on standard Borel spaces. It develops kernel identifiability foundations via kernel-topology notions and proves continuity/robustness of the optimal cost under kernel convergence, enabling reliable learning-driven control. Two data-driven identifiability routes are provided: Bayesian identifiability with finite-time convergence and empirical identifiability via quantization and empirical occupation measures, each leading to near-optimal adaptive policies. It introduces Bayesian and empirical adaptive control schemes, including alternating and simultaneous exploration-exploitation, and an computable empirical learning algorithm with provable convergence to optimality. The framework supports robust, nonparametric learning and control, useful for applications where model structure is unknown and regularity is limited, underpinned by the Young topology and weak convergence tools.
Abstract
We consider an average cost stochastic control problem with standard Borel spaces and an unknown transition kernel. We do not assume a parametric structure on the unknown kernel. We present topologies on kernels which lead to their identifiability and ensures near optimality through robustness to identifiability errors. Following this, we present two data-driven identifiability results; the first one being Bayesian (with finite time convergence guarantees) and the second one empirical (with asymptotic convergence guarantees). The identifiability results are, then, used to design near-optimal adaptive control policies which alternate between periods of exploration, where the agent attempts to learn the true kernel, and periods of exploitation where the knowledge accrued throughout the exploration phase is used to minimize costs. We will establish that such policies are near optimal. Moreover, we will show that near optimality can also be achieved through policies that simultaneously explore and exploit the environment. Thus, our primary contributions are to present very general conditions ensuring identifiability, as well as adaptive learning and control which leads to optimality. Key tools facilitating our analysis are a general measurability theorem, robustness to model learning, and continuity of expected average cost in stationary policies under Young topology.