Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space

Swaminathan S K, Aritra Hazra

TL;DR

SPAARS is introduced, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck.

Abstract

Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.

SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space

TL;DR

SPAARS is introduced, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck.

Abstract

Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.
Paper Structure (31 sections, 7 theorems, 19 equations, 4 figures, 2 tables)

This paper contains 31 sections, 7 theorems, 19 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline-to-Online Reinforcement Learning
  4. Latent Action Spaces and Skill Discovery
  5. Abstract Exploration and Refined Exploitation Framework
  6. Preliminaries and Notations
  7. Gradient Variance Reduction in Latent Space
  8. The Exploitation Gap
  9. The SPAARS Curriculum
  10. SPAARS-SUPE: Instantiation with Temporal Skills
  11. Advantage-Gated Mode Selection
  12. Formal Characterizations: Assumptions, Guarantees, and Tradeoffs
  13. Experimental Evaluation
  14. Setup.
  15. Kitchen-Mixed-v0: Manipulation with Temporal Skills
  16. ...and 16 more sections

Key Result

Proposition 1

Let $\pi_z(z \mid s) = \mathcal{N}(\mu_\theta(s), \sigma^2 I_k)$ and $\pi_a(a \mid s) = \mathcal{N}(\nu_\theta(s), \sigma^2 I_d)$ be isotropic Gaussian policies in latent and raw action spaces respectively. The variance of the REINFORCE gradient estimator satisfies:

Figures (4)

  • Figure 1: Normalized return vs. environment steps on kitchen-mixed-v0. SPAARS-SUPE (gate) warm-starts from the pretrained OPAL IQL policy, reaching SUPE's asymptotic performance 5$\times$ faster and surpassing it by 300k steps.
  • Figure 2: AntMaze-Medium (left): SPAARS-SUPE matches native SUPE performance. Advantage Gate Firing (right): Mean advantage of raw policy over latent policy increases as training progresses, enabling precise goal-reaching.
  • Figure 3: Gate firing patterns over training. Red crosses indicate states where the Advantage Gate selects $\pi_{\mathrm{raw}}$. As training progresses, the gate successfully isolates raw-policy execution to the goal region while utilizing the latent policy for general maze exploration.
  • Figure 4: Normalized return vs. environment steps on hopper-medium-v2 (left) and walker2d-medium-v2 (right). Standalone SPAARS exceeds the offline IQL baseline on both tasks, validating that a CVAE trained on unordered $(s,a)$ pairs provides an effective latent manifold for online fine-tuning. Shaded regions indicate 95% confidence intervals over 3 seeds.

Theorems & Definitions (21)

  • Remark 1: BC Dataset Sufficiency
  • Proposition 1: Variance Reduction Lemma
  • proof
  • Remark 2: Gradient Projection
  • Proposition 2: Exploitation Gap Bound
  • proof
  • Remark 3: Dataset Quality
  • Remark 4: Worst-Case Alternative
  • Definition 1: RND Plateau
  • Proposition 3: Calibration Stability Lemma
  • ...and 11 more