Primal-Dual Sample Complexity Bounds for Constrained Markov Decision Processes with Multiple Constraints
Max Buckley, Konstantinos Papathanasiou, Andreas Spanopoulos
TL;DR
The paper tackles learning near-optimal policies for constrained Markov decision processes with multiple constraints ($d>1$) when transition dynamics are unknown but a generative model is available. It builds a model-based primal-dual framework by forming an empirical CMDP with a perturbed reward and solves a saddle-point problem via iterative updates to a policy and Lagrange multipliers, yielding a mixture policy. The authors derive explicit sample complexity bounds: for the relaxed feasibility setting, $N=\tilde{O}\left( \frac{d|\mathcal{S}| |\mathcal{A}| \log(1/\delta)}{(1-\gamma)^3 \epsilon^2} \right)$, and for the strict feasibility setting, $N=\tilde{O}\left( \frac{d^3 |\mathcal{S}| |\mathcal{A}| \log(1/\delta)}{(1-\gamma)^5 \epsilon^2 {\zeta_{\mathbf{c}}^*}^2} \right)$, where $\zeta_{\mathbf{c}}^*$ is a Slater-type margin. The analysis relies on an $\iota$-gap concentration framework, discretization of dual variables via an $\epsilon_1$-net, and concentration bounds for data-dependent policies. These results provide rigorous guarantees for learning under multiple constraints using a generative-model setting, with practical impact for safe and feasible decision-making in complex environments.
Abstract
This paper addresses the challenge of solving Constrained Markov Decision Processes (CMDPs) with $d > 1$ constraints when the transition dynamics are unknown, but samples can be drawn from a generative model. We propose a model-based algorithm for infinite horizon CMDPs with multiple constraints in the tabular setting, aiming to derive and prove sample complexity bounds for learning near-optimal policies. Our approach tackles both the relaxed and strict feasibility settings, where relaxed feasibility allows some constraint violations, and strict feasibility requires adherence to all constraints. The main contributions include the development of the algorithm and the derivation of sample complexity bounds for both settings. For the relaxed feasibility setting we show that our algorithm requires $\tilde{\mathcal{O}} \left( \frac{d |\mathcal{S}| |\mathcal{A}| \log(1/δ)}{(1-γ)^3ε^2} \right)$ samples to return $ε$-optimal policy, while in the strict feasibility setting it requires $\tilde{\mathcal{O}} \left( \frac{d^3 |\mathcal{S}| |\mathcal{A}| \log(1/δ)}{(1-γ)^5ε^2{ζ_{\mathbf{c}}^*}^2} \right)$ samples.