TraCeS: Trajectory Based Credit Assignment From Sparse Safety Feedback
Siow Meng Low, Akshat Kumar
TL;DR
This work addresses safe reinforcement learning when the true safety criteria are unknown and sparsely labeled. It introduces TraCeS, a trajectory-based credit assignment framework that learns per-timestep safety credits via a Safety Summary Vector, enabling a reformulated constrained RL objective solvable with PPO-Lagrangian. The approach relaxes the need for known costs and budgets and demonstrates strong performance across diverse continuous-control tasks, closely matching oracle baselines while requiring fewer labeled trajectories than prior methods. The result is a scalable, data-efficient method for enforcing safety in unknown-constrained RL settings with practical impact for real-world safety-critical systems.
Abstract
In safe reinforcement learning (RL), auxiliary safety costs are used to align the agent to safe decision making. In practice, safety constraints, including cost functions and budgets, are unknown or hard to specify, as it requires anticipation of all possible unsafe behaviors. We therefore address a general setting where the true safety definition is unknown, and has to be learned from sparsely labeled data. Our key contributions are: first, we design a safety model that performs credit assignment to estimate each decision step's impact on the overall safety using a dataset of diverse trajectories and their corresponding binary safety labels (i.e., whether the corresponding trajectory is safe/unsafe). Second, we illustrate the architecture of our safety model to demonstrate its ability to learn a separate safety score for each timestep. Third, we reformulate the safe RL problem using the proposed safety model and derive an effective algorithm to optimize a safe yet rewarding policy. Finally, our empirical results corroborate our findings and show that this approach is effective in satisfying unknown safety definition, and scalable to various continuous control tasks.