Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anytime-Constrained Reinforcement Learning

Jeremy McMahan, Xiaojin Zhu

TL;DR

The paper addresses the challenge of optimizing policies in anytime-constrained cMDPs, where costs must remain within a budget at every time step with probability 1. It shows that traditional Markovian policies fail and introduces an augmentation of the state with cumulative costs, enabling a fixed-parameter tractable reduction to unconstrained MDPs for planning and learning when cost precision is logarithmic in size. It establishes NP-hardness for exact or non-trivial approximate solutions and then develops additive and relative approximation schemes that efficiently compute or learn approximately feasible policies with optimal value under polynomially bounded max costs, with extensions to various budget and constraint settings. The work also provides a detailed complexity analysis based on cost diversity and demonstrates practical performance on hard instances, offering a principled approach to synthesis under strict, time-localized budgets.

Abstract

We introduce and study constrained Markov Decision Processes (cMDPs) with anytime constraints. An anytime constraint requires the agent to never violate its budget at any point in time, almost surely. Although Markovian policies are no longer sufficient, we show that there exist optimal deterministic policies augmented with cumulative costs. In fact, we present a fixed-parameter tractable reduction from anytime-constrained cMDPs to unconstrained MDPs. Our reduction yields planning and learning algorithms that are time and sample-efficient for tabular cMDPs so long as the precision of the costs is logarithmic in the size of the cMDP. However, we also show that computing non-trivial approximately optimal policies is NP-hard in general. To circumvent this bottleneck, we design provable approximation algorithms that efficiently compute or learn an arbitrarily accurate approximately feasible policy with optimal value so long as the maximum supported cost is bounded by a polynomial in the cMDP or the absolute budget. Given our hardness results, our approximation guarantees are the best possible under worst-case analysis.

Anytime-Constrained Reinforcement Learning

TL;DR

The paper addresses the challenge of optimizing policies in anytime-constrained cMDPs, where costs must remain within a budget at every time step with probability 1. It shows that traditional Markovian policies fail and introduces an augmentation of the state with cumulative costs, enabling a fixed-parameter tractable reduction to unconstrained MDPs for planning and learning when cost precision is logarithmic in size. It establishes NP-hardness for exact or non-trivial approximate solutions and then develops additive and relative approximation schemes that efficiently compute or learn approximately feasible policies with optimal value under polynomially bounded max costs, with extensions to various budget and constraint settings. The work also provides a detailed complexity analysis based on cost diversity and demonstrates practical performance on hard instances, offering a principled approach to synthesis under strict, time-localized budgets.

Abstract

We introduce and study constrained Markov Decision Processes (cMDPs) with anytime constraints. An anytime constraint requires the agent to never violate its budget at any point in time, almost surely. Although Markovian policies are no longer sufficient, we show that there exist optimal deterministic policies augmented with cumulative costs. In fact, we present a fixed-parameter tractable reduction from anytime-constrained cMDPs to unconstrained MDPs. Our reduction yields planning and learning algorithms that are time and sample-efficient for tabular cMDPs so long as the precision of the costs is logarithmic in the size of the cMDP. However, we also show that computing non-trivial approximately optimal policies is NP-hard in general. To circumvent this bottleneck, we design provable approximation algorithms that efficiently compute or learn an arbitrarily accurate approximately feasible policy with optimal value so long as the maximum supported cost is bounded by a polynomial in the cMDP or the absolute budget. Given our hardness results, our approximation guarantees are the best possible under worst-case analysis.
Paper Structure (62 sections, 21 theorems, 43 equations, 6 figures, 5 algorithms)

This paper contains 62 sections, 21 theorems, 43 equations, 6 figures, 5 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Our Contributions.
  3. Related Work.
  4. Knapsack Constraints.
  5. Almost Sure Constraints.
  6. ANYTIME CONSTRAINTS
  7. Constrained Markov Decision Processes.
  8. Interaction Protocol.
  9. Objective.
  10. Optimal Solutions.
  11. Intractability.
  12. FPT REDUCTION
  13. Augmented States.
  14. Augmented Policies.
  15. Analysis.
  16. ...and 47 more sections

Key Result

Proposition 1

Any class of policies that excludes the full cost history is suboptimal for anytime-constrained cMDPs. In particular, Markovian policies can be arbitrarily suboptimal even for cMDPs with $S = 1$ and $A = H = 2$.

Figures (6)

  • Figure 1: Value comparison of our relative approximation and feasibility scheme.
  • Figure 2: Running time of our relative approximation.
  • Figure 3: Our Approximation Scheme vs Our Exact Method on hard cMDP instances.
  • Figure 4: Our No-Violation Scheme vs Our Exact Method on hard cMDP instances.
  • Figure 5: Our No-Violation Scheme vs Our Approximation Scheme on hard cMDP instances.
  • ...and 1 more figures

Theorems & Definitions (55)

  • Proposition 1
  • Corollary 1
  • Proposition 2
  • Lemma 1: Derandomization
  • Theorem 1: Hardness
  • Theorem 2: Hardness of Approximation
  • Remark 1
  • Definition 1: Augmented States
  • Lemma 2
  • Definition 2: Augmented MDP
  • ...and 45 more