SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning

Abstract

Modern deep reinforcement learning (DRL) methods have made significant advances in handling continuous action spaces. However, real-world control systems, especially those requiring precise and reliable performance, often demand interpretability in the sense of a-priori assessments of agent behavior to identify safe or failure-prone interactions with environments. To address this limitation, this work proposes SALSA-RL (Stability Analysis in the Latent Space of Actions), a novel RL framework that models control actions as dynamic, time-dependent variables evolving within a latent space. By employing a pre-trained encoder-decoder and a state-dependent linear system, this approach enables interpretability through local stability analysis, where instantaneous growth in action-norms can be predicted before their execution. It is demonstrated that SALSA-RL can be deployed in a non-invasive manner for assessing the local stability of actions from pretrained RL agents without compromising on performance across diverse benchmark environments. By enabling a more interpretable analysis of action generation, SALSA-RL provides a powerful tool for advancing the design, analysis, and theoretical understanding of RL systems.

Figures (13)

• Figure 1: Overview of the SALSA-RL framework. Our proposed augmentation to pre-trained RL algorithms relies on a latent action representation governed by a time-varying linear dynamical system through a state-conditioned matrix $\mathbf{A}_t$. This enables local stability analyses in the action-state phase for reliable and interpretable RL deployments. The framework integrates seamlessly with existing RL algorithms, maintaining competitive performance. The contours represent the dynamically changing spectral radius of the latent linear system, with the policy seeking bounded regions (in white) with high local stability. Consequently, initializing controllers in regions outside these regions leads to high-risk behavior and potential failure.
• Figure 2: Overview of the SALSA-RL framework. The architecture augments pre-trained RL agents by modeling control actions within a latent space governed by a state-conditioned linear dynamical system, $\mathbf{A}_t$. The contours visualize the time-varying spectral radius $\rho(\mathbf{A}_t)$, serving as a metric for local stability analysis that identifies stable and unstable regions within the action-state phase.
• Figure 3: Local stability analysis of Pendulum control. The trajectory (red) transitions from initial regions of growth ($\rho > 1$) into locally contractive zones ($\rho < 1$), maintaining the system within stable bounds defined by the $\rho=1$ boundary.
• Figure 4: Local stability analysis of CartPole control. Frequent excursions into regions with $\rho(\mathbf{A}_t) > 1$ (red areas) reflect the oscillatory corrective actions required to maintain the pole's upright position.
• Figure 5: Local stability analysis of LunarLander hovering control across different initializations. $\rho=1$ (white line) delineates locally stable and unstable regions. Trajectories and corresponding eigenvalue distributions demonstrate that the stability contours provide early indicators of potential failure, as observed in the unstable Case 2 before the crash.