Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning

Yuxiao Lu, Arunesh Sinha, Pradeep Varakantham

TL;DR

This work tackles safety in temporally extended reinforcement learning by addressing long-horizon tasks with complex safety constraints. It introduces CoSHRL, a two-level framework combining a lower-level distributional RL agent that learns local reward and cost distributions with an upper-level constrained search via ConstrainedRRT* to plan sequences of waypoints under CVaR-based cost constraints. The main contributions include (i) a scalable constrained search method suitable for long-horizon tasks, (ii) the ability to handle rich percentile cost constraints, (iii) rapid policy updates when constraint thresholds or start/goals change, and (iv) a theoretical guarantee of asymptotic optimality under constraints. Empirical results across 2D navigation and image-based Safe-ViZDoom domains show CoSHRL achieving high success rates and CVaR-compliant costs, outperforming several baselines and demonstrating robustness without retraining for varying safety thresholds.

Abstract

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks. In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as robots cleaning different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Search with Hierarchical Reinforcement Learning (CoSHRL) mechanism that combines an upper level constrained search agent (which computes a reward maximizing policy from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoSHRL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR) and can adjust to flexible constraint thresholds without retraining. We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading approaches in constrained and hierarchical RL.

Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning

TL;DR

This work tackles safety in temporally extended reinforcement learning by addressing long-horizon tasks with complex safety constraints. It introduces CoSHRL, a two-level framework combining a lower-level distributional RL agent that learns local reward and cost distributions with an upper-level constrained search via ConstrainedRRT* to plan sequences of waypoints under CVaR-based cost constraints. The main contributions include (i) a scalable constrained search method suitable for long-horizon tasks, (ii) the ability to handle rich percentile cost constraints, (iii) rapid policy updates when constraint thresholds or start/goals change, and (iv) a theoretical guarantee of asymptotic optimality under constraints. Empirical results across 2D navigation and image-based Safe-ViZDoom domains show CoSHRL achieving high success rates and CVaR-compliant costs, outperforming several baselines and demonstrating robustness without retraining for varying safety thresholds.

Abstract

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks. In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as robots cleaning different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Search with Hierarchical Reinforcement Learning (CoSHRL) mechanism that combines an upper level constrained search agent (which computes a reward maximizing policy from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoSHRL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR) and can adjust to flexible constraint thresholds without retraining. We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading approaches in constrained and hierarchical RL.
Paper Structure (6 sections, 2 theorems, 4 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 6 sections, 2 theorems, 4 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Approach
  4. Experiments
  5. Discussion
  6. Acknowledgments

Key Result

Lemma 1

Given $S \subset \mathbb{R}^d$, assuming $-V$ gives the obstacle avoiding shortest path length, $-V$ is a distance metric.

Figures (4)

  • Figure 1: Overview of CoSHRL. Step 1: Train a local goal-conditioned RL agent using multiple randomly selected (o, g) (o is start, g is goal) pairs in a constrained environment (top part). The red square indicates a high-cost region. The learning is local and hence the goal will be unreachable if it's not "near" to the start. In this step, the local value function $V$ and the cost function $V_c$ are learned. Step 2: Generate waypoints guided by $V$ and $V_c$ using the proposed ConstrainedRRT* algorithm (i) The search samples state C, and O is not within the dashed circle of "near" states. Although both A and B are within the circle, the path from O to C via B is better as $V$(O,B) $+ V$(B,C) $< V$(O,A) $+ V$(A,C) using low-level agent's $V$ function. So, edge (B, C) is added to the tree. (ii) For new sample E, E is "near" from C and D, but the edge (C, E) is not valid because of cost constraint $CVaR_{\alpha}(\mathbf{V}_c$(O, B) $+ \mathbf{V}_c$(B, C) $+ \mathbf{V}_c$(C, E)$) > K$. (iii) A path (O, B, C, D, E, L, G) within the cost constraint is found. (iv) As the number of sampled states increases, a better path (O, B, I, D, E, L, G) is found. Step 3: Leveraging the waypoints from step 2, the pre-trained goal-conditioned RL agent completes the task.
  • Figure :
  • Figure :
  • Figure :

Theorems & Definitions (4)

  • Lemma 1
  • Definition 1: Asymptotic optimality karaman2011sampling
  • Definition 2: Asymptotic optimality with constraints
  • Theorem 1