Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction

TL;DR

The proposed Absolute State-wise Constrained Policy Optimization (ASCPO) is a novel general-purpose policy search algorithm that guarantees high-probability state-wise constraint satisfaction for stochastic systems and significantly outperforms existing methods in handling state-wise constraints.

Abstract

Enforcing state-wise safety constraints is critical for the application of reinforcement learning (RL) in real-world problems, such as autonomous driving and robot manipulation. However, existing safe RL methods only enforce state-wise constraints in expectation or enforce hard state-wise constraints with strong assumptions. The former does not exclude the probability of safety violations, while the latter is impractical. Our insight is that although it is intractable to guarantee hard state-wise constraints in a model-free setting, we can enforce state-wise safety with high probability while excluding strong assumptions. To accomplish the goal, we propose Absolute State-wise Constrained Policy Optimization (ASCPO), a novel general-purpose policy search algorithm that guarantees high-probability state-wise constraint satisfaction for stochastic systems. We demonstrate the effectiveness of our approach by training neural network policies for extensive robot locomotion tasks, where the agent must adhere to various state-wise safety constraints. Our results show that ASCPO significantly outperforms existing methods in handling state-wise constraints across challenging continuous control tasks, highlighting its potential for real-world applications.

Figures (23)

• Figure 1: Explanation of ASCPO principles. For simplicity, the distribution is assumed to be Gaussian. Green and blue represent the maximum and expectation of maximum state-wise cost distribution respectively. Y-axis represents the distribution of maximum state-wise cost samples (introduced in \ref{['sec: preliminaries']}) across different initial states. ASCPO is designed to constrain the maximum state-wise cost under safety threshold ($w_i$) while SCPO only focuses on constraining the expectation of state-wise cost and CPO achiam2017constrained lacks the capability to ensure state-wise safety.
• Figure 2: Intuition of the maximum state-wise cost. The evolution of the maximum state-wise cost across a single episode is shown in the blue bars. The red curves represent the state-wise cost, while the green bars indicate the maximum state-wise cost increment at each step.
• Figure 3: Explanation of MV and VM. Since maximum state-wise cost from different start states belong to a mixture of one-dimensional distributions, the variance of maximum state-wise cost can be deconstructed into two components: MeanVariance (MV) and VarianceMean (VM).
• Figure 4: $V_{[D_i]\pi}(\hat{s})$ target of five sampled episodes.
• Figure 5: Robots of continuous control tasks benchmark GUARD.

Theorems & Definitions (10)

• Remark 1
• Definition 1: Upper Probability Bound of Constraint Satisfaction
• Proposition 1
• Remark 2
• Proposition 2
• Proposition 3: Bound of MeanVariance
• Proposition 4: Bound of VarianceMean
• Theorem 1: High Probability State-wise Constraints Satisfaction
• Theorem 2: Monotonic Improvement of Performance
• Proposition 5