Enhance the Safety in Reinforcement Learning by ADRC Lagrangian Methods

TL;DR

This work tackles safety in reinforcement learning by integrating Active Disturbance Rejection Control (ADRC) with Lagrangian methods to regulate constraint satisfaction in CMDPs. By viewing Safe RL as a closed-loop system and introducing an Extended State Observer to estimate disturbances $\hat{f}$, the approach compensates for nonstationarity and noise, reducing phase lag and oscillations relative to classical integral or PID updates. The authors prove that classical and PID Lagrangian updates are special cases of the ADRC framework, derive a principled lower bound on the observer gain $\omega_o$, and provide frequency-domain guarantees of improved disturbance rejection. Empirically, ADRC-Lagrangian variants on OmniSafe benchmarks achieve substantial reductions in constraint violations (up to 74%), violation magnitudes (up to 89%), and average costs (up to 67%), while maintaining competitive rewards, highlighting its potential for robust Safe RL in complex environments.

Abstract

Safe reinforcement learning (Safe RL) seeks to maximize rewards while satisfying safety constraints, typically addressed through Lagrangian-based methods. However, existing approaches, including PID and classical Lagrangian methods, suffer from oscillations and frequent safety violations due to parameter sensitivity and inherent phase lag. To address these limitations, we propose ADRC-Lagrangian methods that leverage Active Disturbance Rejection Control (ADRC) for enhanced robustness and reduced oscillations. Our unified framework encompasses classical and PID Lagrangian methods as special cases while significantly improving safety performance. Extensive experiments demonstrate that our approach reduces safety violations by up to 74%, constraint violation magnitudes by 89%, and average costs by 67\%, establishing superior effectiveness for Safe RL in complex environments.

Figures (21)

• Figure 1: The training curves of PPO with various Lagrangian methods (denoted as CPPOLag, CPPOPID, CPPOADRC) across different tasks, showing episodic returns and costs over five random seeds. Solid lines represent mean values, while shaded areas denote variance. CPPOADRC demonstrates a shorter phase and lower costs compared to baselines, while achieving competitive rewards. Additional results are provided in Appendix \ref{['sec:More_results']}.
• Figure 2: Illustration of the three distinct agents used in our experiments. Car: A simple wheeled agent with low degrees of freedom. Racecar: A dynamic and agile wheeled agent with higher motion complexity. Ant: A multi-legged bionic agent with high degrees of freedom and non-linear dynamics. These agents represent diverse physical characteristics, allowing us to comprehensively evaluate the performance of our method under various physical dynamics.
• Figure 3: Four different tasks used in our experiments. (a) Goal Task: The agent must reach the goal area (blue sphere) without entering dangerous zones (red circles). (b) Button Task: The agent must press the correct button (green) and avoid pressing wrong ones (yellow, purple, etc.) or colliding with gremlins. (c) Push Task: The agent must push the box to the goal location (green circle) while avoiding hazards (red). (d) Circle Env: The agent moves around a simple circular track.
• Figure 4: Training curves of Racecargoal task.
• Figure 5: The training curves of AntButton with various Lagrangian methods across different algorithms.

Theorems & Definitions (11)

• Proposition 4.1
• Theorem 4.2
• proof
• proof : Sketch
• Lemma 3.5: Finite-horizon high-probability envelope for disturbance variation
• proof : Sketch
• Lemma 3.6: ESO estimation-error bound
• Theorem 3.7: ISS-type tracking bound
• Theorem 3.8: Bounded time-average population violation
• proof : Sketch