Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning

TL;DR

The paper tackles safety in constrained offline reinforcement learning under distributional shift between fixed data and the learned policy, where interpolation and extrapolation errors can lead to unsafe decisions. It introduces SP-cdice, a sparsity-based safe conservatism method that uses K-means clustering to identify data-sparse regions and applies a nonuniform cost penalty, avoiding bi-level optimization. Empirical results on discrete Random CMDP and continuous CartPole tasks show SP-cdice achieving competitive or superior returns under cost constraints with reduced computational burden, indicating strong practical promise as a preprocessing step. Overall, SP-cdice provides an efficient, scalable approach to enforce conservatism in offline RL by leveraging data sparsity to balance safety and performance.

Abstract

Reinforcement Learning (RL) has made notable success in decision-making fields like autonomous driving and robotic manipulation. Yet, its reliance on real-time feedback poses challenges in costly or hazardous settings. Furthermore, RL's training approach, centered on "on-policy" sampling, doesn't fully capitalize on data. Hence, Offline RL has emerged as a compelling alternative, particularly in conducting additional experiments is impractical, and abundant datasets are available. However, the challenge of distributional shift (extrapolation), indicating the disparity between data distributions and learning policies, also poses a risk in offline RL, potentially leading to significant safety breaches due to estimation errors (interpolation). This concern is particularly pronounced in safety-critical domains, where real-world problems are prevalent. To address both extrapolation and interpolation errors, numerous studies have introduced additional constraints to confine policy behavior, steering it towards more cautious decision-making. While many studies have addressed extrapolation errors, fewer have focused on providing effective solutions for tackling interpolation errors. For example, some works tackle this issue by incorporating potential cost-maximizing optimization by perturbing the original dataset. However, this, involving a bi-level optimization structure, may introduce significant instability or complicate problem-solving in high-dimensional tasks. This motivates us to pinpoint areas where hazards may be more prevalent than initially estimated based on the sparsity of available data by providing significant insight into constrained offline RL. In this paper, we present conservative metrics based on data sparsity that demonstrate the high generalizability to any methods and efficacy compared to using bi-level cost-ub-maximization.

Figures (4)

• Figure 1: The estimation error, $C_{\text{true}}/C_{\text{est}}$, was assessed for two different data volumes, and the discrepancy, $C_{\text{true}} - C_{\text{est}}$, is depicted as a blue surface. The color gradient below the plot indicates the level of penalty for the corresponding state-action pairs, with specific values for conservatism shown in the adjacent bar. Panel (a) shows the estimation error with a limited number of trajectories $\mathcal{N}$, and panel (b) demonstrates the scenario with a moderate number of trajectories, both yielding errors.
• Figure 2: This figure illustrates the positional state spaces $\mathcal{S} \subset \mathbb{R}^{3}$ of the Cartpole environment. (a) shows the state data distribution, while part (b) displays penalties for state clusters with 10 clusters. Reddish tones indicate higher anti-confidence levels, and little dots are centroids of each cluster. Parts (c) and (d) provide more detailed analyses with $K$ = 50 and 100, respectively.
• Figure 3: A random CMDP with a cost limit of $c=0.1$ was tested using 10 seeds, varying the number of trajectories to compose datasets for better environmental understanding. Two datasets were created: one meeting the cost limit and the other not. The solid line represents the average, and the shaded area shows the standard deviation. Overall, SP-cdice and conservative COptiDICE consistently satisfied the constraint even with fewer trajectories, outperforming other methods in ensuring return and safety. This highlights the effectiveness and efficiency of our novel safety conservatism approach compared to traditional methods.
• Figure 4: We experimented with our algorithm in the continuous domain (Cartpole). Parameter $\alpha$ was chosen for the naive penalty method to ensure compliance with the cost limit. Our approach outperformed or same as non-conservative methods in terms of performance while adhering to the cost limit. In contrast, the naively constrained algorithm achieved lower returns, underscoring the effectiveness of our novel relative conservatism approach using sparsity measures. This demonstrates the applicability and superiority of our method in continuous settings compared to existing approaches.