Stacked Universal Successor Feature Approximators for Safety in Reinforcement Learning

TL;DR

This work introduces SUSFAS, a stacked universal successor feature approximator designed for safety-aware reinforcement learning in continuous control. By learning independent successor features for each objective component and integrating with SAC and a runtime assurance (RTA) controller, SUSFAS improves secondary objectives such as fuel efficiency while maintaining primary task performance. Key contributions include expert stacking (learning SFs independently), extensive ablations against collapsed USFA, and demonstrations that RTA presence is crucial for realizing SUSFAS gains, with substantial fuel-efficiency improvements in mission-critical tasks. The results highlight the potential of stacking successor features to encode safety-controller behaviors and enable robust, multi-objective policies in safety-critical domains.

Abstract

Real-world problems often involve complex objective structures that resist distillation into reinforcement learning environments with a single objective. Operation costs must be balanced with multi-dimensional task performance and end-states' effects on future availability, all while ensuring safety for other agents in the environment and the reinforcement learning agent itself. System redundancy through secondary backup controllers has proven to be an effective method to ensure safety in real-world applications where the risk of violating constraints is extremely high. In this work, we investigate the utility of a stacked, continuous-control variation of universal successor feature approximation (USFA) adapted for soft actor-critic (SAC) and coupled with a suite of secondary safety controllers, which we call stacked USFA for safety (SUSFAS). Our method improves performance on secondary objectives compared to SAC baselines using an intervening secondary controller such as a runtime assurance (RTA) controller.

Figures (5)

• Figure 1: Our proposed architecture, with (a) a detailed SUSFAS architecture block and (b) the overall SUSFAS architecture stack. The variables $\bm{\tilde{\varsigma}}_{0,h_{\varsigma}}$, $\bm{\tilde{\alpha}}_{0,h_{\alpha}}$, $\bm{\tilde{\omega}}_{0,h_{\omega}}$, and $\bm{\tilde{\psi}}_{0,h_{\psi}}$ indicate the feature embedding of the states, actions, task weights, and combined representation after $h_{\varsigma}$, $h_{\alpha}$, $h_{\omega}$, and $h_{\psi}$ hidden layers respectively. The output corresponds to the successor feature approximation (SFA) for the $0$-th successor feature, $\tilde{\psi}_0$. The $\oplus$ symbol indicates concatenation. Our SUSFAS Architecture Stack (b) shows how the $d$-many SFA blocks are stacked and concatenated to form the successor feature vector $\bm{\tilde{\psi}}$.
• Figure 1: Lunar Lander Secondary Controller
• Figure 2: RQ2 -- Investigating effect of RTA on for (a) total fuel usage on Lunar Lander, and (b) $\Delta V$ usage on Inspection3D with specialist and generalist SAC-S and SUSFAS agents. Shaded regions denote the 95% confidence interval.
• Figure 3: RQ2 -- Inspection3D $\Delta V$ usage with RTA ablations.
• Figure 4: RQ3 -- Inspection3D $\Delta V$ usage with varying generalist training ranges.