Conservative Distributional Reinforcement Learning with Safety Constraints

TL;DR

A novel off-policy reinforcement learning algorithm called Conservative Distributional Maximum a Posteriori Policy Optimization (CDMPO) is presented, which uses a conservative value function loss to reduce the number of violations of constraints during the exploration process.

Abstract

Safety exploration can be regarded as a constrained Markov decision problem where the expected long-term cost is constrained. Previous off-policy algorithms convert the constrained optimization problem into the corresponding unconstrained dual problem by introducing the Lagrangian relaxation technique. However, the cost function of the above algorithms provides inaccurate estimations and causes the instability of the Lagrange multiplier learning. In this paper, we present a novel off-policy reinforcement learning algorithm called Conservative Distributional Maximum a Posteriori Policy Optimization (CDMPO). At first, to accurately judge whether the current situation satisfies the constraints, CDMPO adapts distributional reinforcement learning method to estimate the Q-function and C-function. Then, CDMPO uses a conservative value function loss to reduce the number of violations of constraints during the exploration process. In addition, we utilize Weighted Average Proportional Integral Derivative (WAPID) to update the Lagrange multiplier stably. Empirical results show that the proposed method has fewer violations of constraints in the early exploration process. The final test results also illustrate that our method has better risk control.

Figures (5)

• Figure 1: Comparison of PPO-Lag, TRPO-Lag, CPO, SAC-Lag, WCSAC and CDMPO during training in CarGoal1(top row) and PointGoal1(botton row) . The lines are the average of four runs, and the shaded area is the standard deviation.
• Figure 2: The test performance for the final policies of each algorithm. Every algorithm runs for 100 episodes. Environment: CarGoal1 (top row), PointGoal1 (botton row), cost limit 25.
• Figure 3: Different ablation performances of CDMPO during training. "w/o CDCL" means do not use CDCL loss. Environment: CarGoal1, cost limit 25
• Figure 4: Performance of different algorithms during training, which corresponds to section 5.5(2). Environment: CarGoal1, cost limit 25
• Figure 5: Different types of PID control on the Lagrange multiplier damps oscillations. Blue line cdmpo (means use WAPID), orange line cdmpo-p (means only use proportional control)), green line cdmpo-pi(means use PID). Environment: CarGoal1, cost limit 25