$\mathrm{E^{2}CFD}$: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Model

TL;DR

E^2CFD proposes a cost-function design framework for safe reinforcement learning that uses a large language model to generate initial cost functions, an Error Code Filtering module to ensure correctness, and a Fast Performance Evaluation loop to quickly steer iterations. By framing cost design as a CDP within CMDP, the approach aims to align the optimization objective with varied safety requirements, enabling generalized performance across traditional, zero-violation, and almost-sure constraints. Empirical results on Safety Gym demonstrate that E^2CFD achieves superior or competitive task performance and safety compliance with greater generalizability and efficiency than traditional safe RL methods and manually designed costs. The work highlights the potential of gray-box LLM assistance to reduce design effort and improve interpretability, while identifying prompts and multi-constraint scalability as avenues for future improvement.

Abstract

Different classes of safe reinforcement learning algorithms have shown satisfactory performance in various types of safety requirement scenarios. However, the existing methods mainly address one or several classes of specific safety requirement scenario problems and cannot be applied to arbitrary safety requirement scenarios. In addition, the optimization objectives of existing reinforcement learning algorithms are misaligned with the task requirements. Based on the need to address these issues, we propose $\mathrm{E^{2}CFD}$, an effective and efficient cost function design framework. $\mathrm{E^{2}CFD}$ leverages the capabilities of a large language model (LLM) to comprehend various safety scenarios and generate corresponding cost functions. It incorporates the \textit{fast performance evaluation (FPE)} method to facilitate rapid and iterative updates to the generated cost function. Through this iterative process, $\mathrm{E^{2}CFD}$ aims to obtain the most suitable cost function for policy training, tailored to the specific tasks within the safety scenario. Experiments have proven that the performance of policies trained using this framework is superior to traditional safe reinforcement learning algorithms and policies trained with carefully designed cost functions.

Figures (8)

• Figure 2: Framework of $\mathrm{E^{2}CFD}$.
• Figure 3: Structual diagram of Error Code Filtering(ECF) module.
• Figure 4: Performance on traditional safety requirement scenarios. In this scenario, for the traditional safe RL algorithm, we set $d=10$ for the expected cost limit (black dashed line).
• Figure 5: Performance on zero-violation safety requirement scenarios. In this scenario, for the zero-violation safe RL algorithm, we set $d=0$ for the expected cost limit.
• Figure 6: Performance on almost surely safety requirement scenarios. Box plots show the overall satisfaction of the constraints and the distribution characteristics such as median, 75th and 25th quantiles and outlier points of the experimental results, which are helpful for evaluating whether the constraints are satisfied almost surely.

Theorems & Definitions (1)

• Definition 1: Cost Design Problem