Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Prediction

TL;DR

The paper tackles safety in online model-based RL for systems with unknown dynamics by integrating scalable dynamics modeling (Quadrature Fourier Features) with probabilistic safety guarantees (Adaptive Conformal Prediction-based Control Barrier Functions) and an optimism-driven exploration strategy. The approach yields provable safety and data-efficiency in episodic learning, supported by a regret bound of $\mathbb{E}[R_T]\leq \widetilde{\mathcal{O}}(\sqrt{P(P+n+K)K^3 T})$ and model-agnostic uncertainty handling. The method is validated on a mobile robot and an inverted pendulum, showing collision-free exploration, improved computation compared to GP-based methods, and near-optimal safe performance. These results highlight a practical, model-agnostic path to safe, sample-efficient learning for nonlinear control tasks.

Abstract

Safety is a critical concern in learning-enabled autonomous systems especially when deploying these systems in real-world scenarios. An important challenge is accurately quantifying the uncertainty of unknown models to generate provably safe control policies that facilitate the gathering of informative data, thereby achieving both safe and optimal policies. Additionally, the selection of the data-driven model can significantly impact both the real-time implementation and the uncertainty quantification process. In this paper, we propose a provably sample efficient episodic safe learning framework that remains robust across various model choices with quantified uncertainty for online control tasks. Specifically, we first employ Quadrature Fourier Features (QFF) for kernel function approximation of Gaussian Processes (GPs) to enable efficient approximation of unknown dynamics. Then the Adaptive Conformal Prediction (ACP) is used to quantify the uncertainty from online observations and combined with the Control Barrier Functions (CBF) to characterize the uncertainty-aware safe control constraints under learned dynamics. Finally, an optimism-based exploration strategy is integrated with ACP-based CBFs for safe exploration and near-optimal safe nonlinear control. Theoretical proofs and simulation results are provided to demonstrate the effectiveness and efficiency of the proposed framework.

Figures (4)

• Figure 1: Computationally and sample efficient safe learning framework. The dynamics by Thompson sampling is embedded into the Model Predictive Path Integral (MPPI)williams2018informationwilliams2017information to generate the reference policy. Then, ACP-based CBF is applied to guarantee safety. The framework is performed episodically.
• Figure 2: Simulation results on a mobile robot with single integrator dynamics. Left: The exploration path by our framework(QFF). $\tau{(\centerdot)}$ represents the episode index $(\centerdot)$ of the exploration path. Right: Ablation study to compare the safety performance from the exploration paths under three methods (dynamics is under the QFF model) respectively, i.e., MPPI, MPPI-CBF, and MPPI-CBF-ACP (ours).
• Figure 3: Computation and sample efficiency evaluation.Left: Training time for the learning process from examples in Fig. \ref{['Fig: integrator']}. GP is not scalable as the number of training episodes increases. The time cost is validated on the i9-10900X CPU and RTX A5000 Graphics Card. Right: Reward for the pendulum under four models (GT-ground truth dynamics). The converge speed of our method is close to GPs and enjoys a better reward.
• Figure 4: Pendulum simulation. Left: Ablation study to compare the safety performance from the exploration paths under three methods (dynamics is under the QFF model), i.e., MPPI, MPPI-CBF, and MPPI-CBF-ACP (ours). Right: Applying the learned dynamics for different initial states.

Theorems & Definitions (2)

• proof
• proof