Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Infinite-Horizon Reach-Avoid Zero-Sum Games via Deep Reinforcement Learning

Jingqi Li, Donggun Lee, Somayeh Sojoudi, Claire J. Tomlin

TL;DR

The theoretical and empirical results suggest that the proposed method could learn reliably the reach-avoid set and the optimal control policy even with neural network approximation.

Abstract

In this paper, we consider the infinite-horizon reach-avoid zero-sum game problem, where the goal is to find a set in the state space, referred to as the reach-avoid set, such that the system starting at a state therein could be controlled to reach a given target set without violating constraints under the worst-case disturbance. We address this problem by designing a new value function with a contracting Bellman backup, where the super-zero level set, i.e., the set of states where the value function is evaluated to be non-negative, recovers the reach-avoid set. Building upon this, we prove that the proposed method can be adapted to compute the viability kernel, or the set of states which could be controlled to satisfy given constraints, and the backward reachable set, or the set of states that could be driven towards a given target set. Finally, we propose to alleviate the curse of dimensionality issue in high-dimensional problems by extending Conservative Q-Learning, a deep reinforcement learning technique, to learn a value function such that the super-zero level set of the learned value function serves as a (conservative) approximation to the reach-avoid set. Our theoretical and empirical results suggest that the proposed method could learn reliably the reach-avoid set and the optimal control policy even with neural network approximation.

Infinite-Horizon Reach-Avoid Zero-Sum Games via Deep Reinforcement Learning

TL;DR

The theoretical and empirical results suggest that the proposed method could learn reliably the reach-avoid set and the optimal control policy even with neural network approximation.

Abstract

In this paper, we consider the infinite-horizon reach-avoid zero-sum game problem, where the goal is to find a set in the state space, referred to as the reach-avoid set, such that the system starting at a state therein could be controlled to reach a given target set without violating constraints under the worst-case disturbance. We address this problem by designing a new value function with a contracting Bellman backup, where the super-zero level set, i.e., the set of states where the value function is evaluated to be non-negative, recovers the reach-avoid set. Building upon this, we prove that the proposed method can be adapted to compute the viability kernel, or the set of states which could be controlled to satisfy given constraints, and the backward reachable set, or the set of states that could be driven towards a given target set. Finally, we propose to alleviate the curse of dimensionality issue in high-dimensional problems by extending Conservative Q-Learning, a deep reinforcement learning technique, to learn a value function such that the super-zero level set of the learned value function serves as a (conservative) approximation to the reach-avoid set. Our theoretical and empirical results suggest that the proposed method could learn reliably the reach-avoid set and the optimal control policy even with neural network approximation.
Paper Structure (10 sections, 8 theorems, 43 equations, 7 figures, 1 algorithm)

This paper contains 10 sections, 8 theorems, 43 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. A New Value Function for Infinite-horizon Reach-Avoid Games
  4. Computing the Viability Kernel and the Backward Reachable Set
  5. A Deep Reinforcement Learning Algorithm
  6. Experiments
  7. 2D Experiment for Reach-Avoid Game
  8. 6D Experiment for Reach-Avoid Game
  9. 6D Experiments on Learning Viability Kernel and Backward Reachable Set
  10. Conclusion

Key Result

Theorem 1

Consider the value function $V(x)$ defined in eq:inf_horizon_reach_avoid_problem. For every $\gamma\in[0,1)$, it holds that $\{x\in\mathbb{R}^n:V(x)>0\} = \mathcal{RA(T,C)}$.

Figures (7)

  • Figure 1: The target set and the constraint set in the 2D experiment
  • Figure 2: Comparison of the 2D reach-avoid set learned by tabular Q-learning and Algorithm \ref{['alg:DQN']}. The yellow area corresponds to the reach-avoid set.
  • Figure 3: The avoidance area and the target area in the three-cart reach-avoid zero-sum game experiment
  • Figure 4: Visualization of reach-avoid sets under different CQL penalty weights $\lambda$, with $[x_2,v_2,x_3,v_3]=[-1,1,1, -1]$. In the first row, the yellow area represents the reach-avoid set. In the second row, we plot the value of each point in the corresponding plots in the first row.
  • Figure 5: The control and disturbance policies extracted from the neural network value function learned by Algorithm \ref{['alg:DQN']} with $\lambda = 0.0$. The other four states are $[x_2,v_2,x_3,v_3] = [-1,1,1,-1]$. The yellow and blue areas in the left plot correspond to the control inputs with the values 1 and -1, respectively. The yellow and blue ares in the right plot correspond to the disturbances with the values $0.5$ and $-0.5$, respectively.
  • ...and 2 more figures

Theorems & Definitions (17)

  • Definition 1: Non-anticipative strategy
  • Theorem 1
  • Theorem 2
  • Theorem 3: Lipschitz continuity
  • Remark 1
  • Proposition 1
  • Remark 2
  • Proposition 2
  • Theorem 4
  • proof : Proof of Theorem \ref{['thm:V>0']}
  • ...and 7 more