Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Formalizing the Relationship between Hamilton-Jacobi Reachability and Reinforcement Learning

Prashant Solanki, Isabelle El-Hajj, Jasper van Beers, Erik-Jan van Kampen, Coen de Visser

TL;DR

It is proved that the resultant travel-cost value function is the unique bounded viscosity solution of a time-dependent Hamilton-Jacobi Bellman (HJB) Partial Differential Equation (PDE) with zero terminal data, whose negative sublevel set equals the strict backward-reachable tube.

Abstract

We unify Hamilton-Jacobi (HJ) reachability and Reinforcement Learning (RL) through a proposed running cost formulation. We prove that the resultant travel-cost value function is the unique bounded viscosity solution of a time-dependent Hamilton-Jacobi Bellman (HJB) Partial Differential Equation (PDE) with zero terminal data, whose negative sublevel set equals the strict backward-reachable tube. Using a forward reparameterization and a contraction inducing Bellman update, we show that fixed points of small-step RL value iteration converge to the viscosity solution of the forward discounted HJB. Experiments on a classical benchmark compare learned values to semi-Lagrangian HJB ground truth and quantify error.

Formalizing the Relationship between Hamilton-Jacobi Reachability and Reinforcement Learning

TL;DR

It is proved that the resultant travel-cost value function is the unique bounded viscosity solution of a time-dependent Hamilton-Jacobi Bellman (HJB) Partial Differential Equation (PDE) with zero terminal data, whose negative sublevel set equals the strict backward-reachable tube.

Abstract

We unify Hamilton-Jacobi (HJ) reachability and Reinforcement Learning (RL) through a proposed running cost formulation. We prove that the resultant travel-cost value function is the unique bounded viscosity solution of a time-dependent Hamilton-Jacobi Bellman (HJB) Partial Differential Equation (PDE) with zero terminal data, whose negative sublevel set equals the strict backward-reachable tube. Using a forward reparameterization and a contraction inducing Bellman update, we show that fixed points of small-step RL value iteration converge to the viscosity solution of the forward discounted HJB. Experiments on a classical benchmark compare learned values to semi-Lagrangian HJB ground truth and quantify error.
Paper Structure (18 sections, 18 theorems, 142 equations, 2 figures)

This paper contains 18 sections, 18 theorems, 142 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. System Dynamics
  4. Travel-Cost Value Function
  5. hjb PDE for the Travel-Cost Value Function
  6. Reachability via Running Cost (Strict BRT)
  7. Relative Exponential Discount
  8. Reachability Encoding with Relative Discount (Strict brt)
  9. HJB reachability and RL Equivalence
  10. PDE limit for implementable one-step schemes
  11. Methodology and Experiments
  12. Stage I: Travel--vs--Reach hjb (zero/negative level set equivalence)
  13. Stage II: Forward discounted hjb $\leftrightarrow$ RL with continuation
  14. Evaluation protocol (common to both stages)
  15. Results
  16. ...and 3 more sections

Key Result

Theorem 1

For $(t,x)\in[0,T]\times\mathbb R^n$, let and define Under the standing assumptions, $V$ is a unique and bounded viscosity solution of

Figures (2)

  • Figure 1: Travel- vs. reach-cost hjb solutions computed on $\mathcal{X}_{10}$ for double integrator.
  • Figure 2: Forward discounted hjb $\leftrightarrow$ RL on $\mathcal{X}_{2.5}=[-2.5,2.5]^2$ with $\Delta\tau=0.05$, $\lambda=1.0$ ($\gamma=e^{-0.05}$). Visual agreement is strong across the ROI; quantitative errors are reported in equation \ref{['eq:errors']}.

Theorems & Definitions (20)

  • Theorem 1: hjb characterization; viscosity sense
  • Proposition 1: Negative sublevel equals strict brt
  • Proposition 2: Zero level equals complement
  • Lemma 1: Well-posedness
  • Lemma 2: dpp with relative discount
  • Lemma 3: Boundedness
  • Lemma 4: Lipschitz in state
  • Lemma 5: Time continuity
  • Lemma 6
  • Lemma 7
  • ...and 10 more