Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration

TL;DR

The paper tackles constraint violations in CRL by addressing systematic underestimation of the cost value. It introduces MICE, a memory-driven intrinsic cost mechanism realized through a flashbulb memory of high-cost states and a pseudo-count intrinsic cost, integrated via an extrinsic-intrinsic cost value function with bias correction. The approach is optimized within a trust region, yielding theoretical bounds on constraint violation and convergence guarantees, and is validated across Safety Gym and MuJoCo tasks where it reduces violations while preserving policy performance. This work advances safe exploration in CRL with a principled, memory-informed framework that can be integrated with existing CRL algorithms.

Abstract

Constrained Reinforcement Learning (CRL) aims to maximize cumulative rewards while satisfying constraints. However, existing CRL algorithms often encounter significant constraint violations during training, limiting their applicability in safety-critical scenarios. In this paper, we identify the underestimation of the cost value function as a key factor contributing to these violations. To address this issue, we propose the Memory-driven Intrinsic Cost Estimation (MICE) method, which introduces intrinsic costs to mitigate underestimation and control bias to promote safer exploration. Inspired by flashbulb memory, where humans vividly recall dangerous experiences to avoid risks, MICE constructs a memory module that stores previously explored unsafe states to identify high-cost regions. The intrinsic cost is formulated as the pseudo-count of the current state visiting these risk regions. Furthermore, we propose an extrinsic-intrinsic cost value function that incorporates intrinsic costs and adopts a bias correction strategy. Using this function, we formulate an optimization objective within the trust region, along with corresponding optimization methods. Theoretically, we provide convergence guarantees for the proposed cost value function and establish the worst-case constraint violation for the MICE update. Extensive experiments demonstrate that MICE significantly reduces constraint violations while preserving policy performance comparable to baselines.

Figures (15)

• Figure 1: Structure of MICE. Underestimation of cost value in high-cost regions causes constraint violations. The Flashbulb Memory records high-cost regions by storing previously explored unsafe states. The intrinsic cost is computed through the pseudo-count of the current state's visit to high-cost regions in memory. The trust region ensures the alignment of the stored high-cost regions with the current policy. The extrinsic-intrinsic cost critic mitigates underestimation in high-cost regions.
• Figure 2: Underestimation bias across environments. The x-axis is time steps, the y-axis is the cost value estimate minus the true value, and the dashed line is the zero deviation.
• Figure 3: Comparison of MICE to baselines on Safety Gym. The x-axis is the total number of training steps, the y-axis is the average return or constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
• Figure 4: Comparison of MICE to baselines on Safety MuJoCo. The x-axis is the total number of training steps, the y-axis is the average return or constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line in the cost plot is the constraint threshold which is 25.
• Figure 5: Validation experiments of mitigating underestimation with MICE. The y-axis is the cost value estimate minus the true value, and the dashed line is the zero deviation.

Theorems & Definitions (12)

• Proposition 4.1
• Lemma 4.2
• Theorem 4.3: Extrinsic-intrinsic Constraint Bounds
• Theorem 4.4: MICE Update Worst-Case Constraint Violation
• Theorem 4.5: Convergence Analysis
• Lemma 2.1
• Proposition 2.2
• Lemma 2.3
• Theorem 2.4: Extrinsic-intrinsic Constraint Bounds
• Theorem 2.5: MICE Update Worst-Case Constraint Violation