An Empirical Study of Lagrangian Methods in Safe Reinforcement Learning

TL;DR

This work provides a systematic empirical examination of the Lagrange multiplier $\lambda$ in safe reinforcement learning, focusing on optimality and stability across four Safety Gym tasks. By introducing $\lambda$-profiles, the authors show that the best $\lambda^*$ is highly task-dependent and not generally inferred a priori. They compare fixed $\lambda$, gradient-ascent updates, and PID-controlled updates, finding that automated updates can match or exceed fixed optima in several tasks, though they can induce oscillations that PID control may mitigate only with careful tuning. Overall, the study highlights the sensitivity and instability challenges of Lagrangian methods in Safe RL and calls for further stabilization research, with code made available for reproducibility.

Abstract

In safety-critical domains such as robotics, navigation and power systems, constrained optimization problems arise where maximizing performance must be carefully balanced with associated constraints. Safe reinforcement learning provides a framework to address these challenges, with Lagrangian methods being a popular choice. However, the effectiveness of Lagrangian methods crucially depends on the choice of the Lagrange multiplier $λ$, which governs the trade-off between return and constraint cost. A common approach is to update the multiplier automatically during training. Although this is standard in practice, there remains limited empirical evidence on the robustness of an automated update and its influence on overall performance. Therefore, we analyze (i) optimality and (ii) stability of Lagrange multipliers in safe reinforcement learning across a range of tasks. We provide $λ$-profiles that give a complete visualization of the trade-off between return and constraint cost of the optimization problem. These profiles show the highly sensitive nature of $λ$ and moreover confirm the lack of general intuition for choosing the optimal value $λ^*$. Our findings additionally show that automated multiplier updates are able to recover and sometimes even exceed the optimal performance found at $λ^*$ due to the vast difference in their learning trajectories. Furthermore, we show that automated multiplier updates exhibit oscillatory behavior during training, which can be mitigated through PID-controlled updates. However, this method requires careful tuning to achieve consistently better performance across tasks. This highlights the need for further research on stabilizing Lagrangian methods in safe reinforcement learning. The code used to reproduce our results can be found at https://github.com/lindsayspoor/Lagrangian_SafeRL.

Figures (7)

• Figure 1: Smoothed profiles of the Lagrange multiplier, averaged over 10 seeds. The cost limit is set to 25. The ☆ indicate the return and cost for a manually fixed value of $\lambda$ at the intersection of the cost curve with the cost limit, corresponding to the optimal trade-off $\lambda^*$. Across all tasks, the optimal multiplier value $\lambda^*$ varies and the return decreases for $\lambda > \lambda^*$, showing that only values close to ☆ yield high-return solutions. The dashed horizontal lines indicate the return and cost values at the end of training for the GA-updated multiplier, averaged over the last 5% of training (for the training curves, see Fig. \ref{['fig:comparison_training_all']}). For the Circle1 and Goal1 tasks, the GA-update recovers the same optimum as the manually fixed $\lambda$ (matching the ☆), while for the Button1 and Push1 tasks, it even exceeds the performance achieved with the fixed multiplier.
• Figure 2: Smoothed training curves of the return (top) and cost (bottom) for a GA-updated$\lambda$ and for a manually fixed $\lambda^*$ across training, averaged across 10 seeds. The training curves of the fixed value of $\lambda$ corresponds with the model trained for the optimal value $\lambda^*$, indicated by the ☆ in Fig. \ref{['fig:results_cost_limit_25_all']}. The learning trajectory for the manually fixed $\lambda$ updates more gradually and conservatively as compared to the GA-updated $\lambda$. Note that the training curves for the Push1 task do not show full convergence at the end of the depicted training iterates.
• Figure 3: Smoothed training curves of the Lagrange multiplier (top), return (middle) and cost (bottom) for a GA-updated$\lambda$ and for PID-updated$\lambda$ across training, averaged across 6 seeds. $K_P=10^{-4}, K_I=10^{-4}, K_D=0.0$. The ☆ indicate the point of best return after the cost constraint of 25.0 is first satisfied, considering only those time steps where the cumulative cost remains below the specified cost limit. The GA-update method exhibits more oscillations in the updates in $\lambda$, whereas PID shows more stability throughout the training process.
• Figure B.1: Optimal multiplier value $\lambda^*$ for all 4 evaluated benchmark tasks. The optimal value gradually increases with task complexity, except for the Push1 task, which is considered to be the most complex task out of the four. This shows that there is no general intuition on the underlying relation between the optimal multiplier $\lambda^*$ and the task complexity, and that there is no such thing as a task-agnostic value for $\lambda^*$.
• Figure B.2: Return-cost curves for all 4 benchmark tasks with a GA-updated multiplier, averaged over 10 seeds. The return and cost values for the fixed multiplier $\lambda$ correspond with the last 5% of training iterates, and each data point on the fixed $\lambda$ line corresponds with one of the values $\lambda$ was manually kept fixed at. Each data point on the line for the GA-updated multiplier corresponds with a different cost limit set as a target for the penalty loss term during training, which corresponds to the averaged last 5% of the training curves shown in Fig. \ref{['fig: training_curve_cost_lims']}. Even though these plots illustrate the relationship between two conflicting objectives, none of them exhibit a clear Pareto frontier, which is typically characteristic of multi-objective optimization problems.