Near-Optimal Sample Complexity for Online Constrained MDPs
Chang Liu, Yunfan Li, Lin F. Yang
TL;DR
This paper tackles safe online reinforcement learning under CMDPs by presenting a model-based primal-dual algorithm with doubling batch updates. It provides near-optimal, regime-specific sample complexity bounds: \tilde{O}(SAH^3/\varepsilon^2) for relaxed feasibility (allowing small violations) and \tilde{O}(SAH^5/(\varepsilon^2\zeta^2)) for strict feasibility (zero violations), matching corresponding lower bounds in each setting. The approach leverages a saddle-point formulation with discretized dual variables, optimistic/pessimistic value estimates, and a doubling data-collection scheme to decouple model estimation from value estimation. The results imply online CMDP learning can be as efficient as learning with a generative model and as easy as unconstrained MDP learning when small violations are permitted, marking a significant step in safe, sample-efficient online CMDP algorithms with minimax optimal guarantees.
Abstract
Safety is a fundamental challenge in reinforcement learning (RL), particularly in real-world applications such as autonomous driving, robotics, and healthcare. To address this, Constrained Markov Decision Processes (CMDPs) are commonly used to enforce safety constraints while optimizing performance. However, existing methods often suffer from significant safety violations or require a high sample complexity to generate near-optimal policies. We address two settings: relaxed feasibility, where small violations are allowed, and strict feasibility, where no violation is allowed. We propose a model-based primal-dual algorithm that balances regret and bounded constraint violations, drawing on techniques from online RL and constrained optimization. For relaxed feasibility, we prove that our algorithm returns an $\varepsilon$-optimal policy with $\varepsilon$-bounded violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^3}{\varepsilon^2}\right)$ learning episodes, matching the lower bound for unconstrained MDPs. For strict feasibility, we prove that our algorithm returns an $\varepsilon$-optimal policy with zero violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^5}{\varepsilon^2ζ^2}\right)$ learning episodes, where $ζ$ is the problem-dependent Slater constant characterizing the size of the feasible region. This result matches the lower bound for learning CMDPs with access to a generative model. Our results demonstrate that learning CMDPs in an online setting is as easy as learning with a generative model and is no more challenging than learning unconstrained MDPs when small violations are allowed.