Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constrained Meta Reinforcement Learning with Provable Test-Time Safety

Tingting Ni, Maryam Kamgarpour

TL;DR

This work addresses safe learning across a distribution of CMDPs by proposing a constrained meta-RL framework that leverages offline training to enable safe, rapid adaptation at test time. The method builds a covering CMDP set $\mathcal{U}$ and a simultaneously feasible policy $\pi_s$ during training, and then uses an adaptive mixture of $\pi_s$ with near-optimal candidates from $\mathcal{U}$ during testing to ensure safety while approaching near-optimal rewards. The authors prove high-probability safety guarantees and derive a data-dependent sample complexity bound $\tilde{\mathcal{O}}\left(\xi^{-2}\varepsilon^{-2}(1-\gamma)^{-5}\mathcal{C}_{\xi\varepsilon(1-\gamma)^3}(\mathcal{D},\delta)\right)$, along with a matching problem-dependent lower bound, establishing near-optimality. Empirical results on a gridworld task demonstrate improved learning efficiency and safe exploration compared to constrained RL baselines and a constrained meta-RL baseline, highlighting practical impact for safety-critical robotics and healthcare settings.

Abstract

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure the safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.

Constrained Meta Reinforcement Learning with Provable Test-Time Safety

TL;DR

This work addresses safe learning across a distribution of CMDPs by proposing a constrained meta-RL framework that leverages offline training to enable safe, rapid adaptation at test time. The method builds a covering CMDP set and a simultaneously feasible policy during training, and then uses an adaptive mixture of with near-optimal candidates from during testing to ensure safety while approaching near-optimal rewards. The authors prove high-probability safety guarantees and derive a data-dependent sample complexity bound , along with a matching problem-dependent lower bound, establishing near-optimality. Empirical results on a gridworld task demonstrate improved learning efficiency and safe exploration compared to constrained RL baselines and a constrained meta-RL baseline, highlighting practical impact for safety-critical robotics and healthcare settings.

Abstract

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure the safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.
Paper Structure (55 sections, 12 theorems, 131 equations, 2 figures, 1 table, 4 algorithms)

This paper contains 55 sections, 12 theorems, 131 equations, 2 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Notation:
  4. Constrained Markov Decision Processes
  5. Feasibility of policies for $\mathcal{M}_i$:
  6. Distance between CMDPs:
  7. Constrained Meta Reinforcement Learning
  8. The training phase
  9. Generative model:
  10. The testing phase
  11. Rollout model:
  12. Objective:
  13. Algorithm Design
  14. Training Phase
  15. Testing Phase
  16. ...and 40 more sections

Key Result

Lemma 3.5

Let Assumption ass_slater hold and set $(8L+18)\varepsilon\le \xi$, where $L:=(1-\gamma)^{-1}+2\gamma(1-\gamma)^{-2}$. With probability at least $1 - 25\delta - 9\delta\ln C_\varepsilon(\mathcal{D},\delta)$, the following holds:

Figures (2)

  • Figure 1: (a): Optimal policies and values for noise levels $i=0.1,0.3$. (b): Reward regret and constraint values of our algorithm, Safe Meta-RL, DOPE+, and LB-SGD, averaged over 10 independent runs; in each run a CMDP $\mathcal{M}_i$ is sampled. All plots share the same x-axis representing the iteration number.
  • Figure 2: The hard CMDP instance $\mathcal{M}_0$ from vaswani2022near, adapted to our constrained meta RL setting.

Theorems & Definitions (28)

  • Definition 2.2: Safe exploration
  • Definition 2.3
  • Definition 3.1: CMDP check oracle
  • Definition 3.2: Optimal policy oracle
  • Definition 3.4: Simultaneously feasible policy oracle
  • Lemma 3.5
  • Definition 4.1
  • Theorem 4.2
  • Corollary 4.3
  • Remark 4.4
  • ...and 18 more