Safe CoR: A Dual-Expert Approach to Integrating Imitation Learning and Safe Reinforcement Learning Using Constraint Rewards

TL;DR

Safe CoR addresses the dual need for high performance and stringent safety in autonomous driving by coupling imitation-learning and safe reinforcement learning through a constraint reward framework. It leverages two expert demonstrations—reward-focused and safety-focused—and introduces CoR to guide policy updates via a dual objective: maximize reward while enforcing safety constraints with CoR regularization. The approach is formalized on a CMDP basis and integrates CoR into both the objective and constraint, controlled by small risk-adjustment parameters. Empirical results across Safety Gym, Metadrive, and the Jackal platform demonstrate reduced constraint violations and improved performance, including notable real-world gains via sim-to-real transfer, underscoring the framework’s robustness and practical impact.

Abstract

In the realm of autonomous agents, ensuring safety and reliability in complex and dynamic environments remains a paramount challenge. Safe reinforcement learning addresses these concerns by introducing safety constraints, but still faces challenges in navigating intricate environments such as complex driving situations. To overcome these challenges, we present the safe constraint reward (Safe CoR) framework, a novel method that utilizes two types of expert demonstrations$\unicode{x2013}$reward expert demonstrations focusing on performance optimization and safe expert demonstrations prioritizing safety. By exploiting a constraint reward (CoR), our framework guides the agent to balance performance goals of reward sum with safety constraints. We test the proposed framework in diverse environments, including the safety gym, metadrive, and the real$\unicode{x2013}$world Jackal platform. Our proposed framework enhances the performance of algorithms by $39\%$ and reduces constraint violations by $88\%$ on the real-world Jackal platform, demonstrating the framework's efficacy. Through this innovative approach, we expect significant advancements in real-world performance, leading to transformative effects in the realm of safe and reliable autonomous agents.

Figures (4)

• Figure 1: Experiment environments
• Figure 2: Safety gym results. The cost rate refers to the average cost per step and the dashed line indicates the constraint threshold.
• Figure 3: Snapshots of OffTRC (a) and OffTRC with safe CoR (b) after training in metadrive simulator
• Figure 4: Snapshots of CPO (a) and CPO with safe CoR (b) in the real-world Jackal platform.