Value Approximation for Two-Player General-Sum Differential Games with State Constraints

TL;DR

This work tackles value function approximation for two-player general-sum differential games with state constraints by solving Hamilton-Jacobi-Isaacs PDEs with physics-informed neural networks. It identifies discontinuities caused by state constraints and proposes three remedies—hybrid learning, value hardening, and epigraphical learning—to enable scalable, safe decision-making in high-dimensional settings. Empirical results across 5D vehicle, 9D road/drone, and 13D drone scenarios show that hybrid learning, which combines supervisory equilibria with PINN-based learning of costates, achieves superior generalization and safety compared to vanilla PINN, supervised learning, and the other two approaches. The paper also extends the epigraphical technique to general-sum games with state constraints and analyzes safety considerations, the consistency between BVP and HJI values, and the critical role of costate loss in safety performance.

Abstract

Solving Hamilton-Jacobi-Isaacs (HJI) PDEs numerically enables equilibrial feedback control in two-player differential games, yet faces the curse of dimensionality (CoD). While physics-informed neural networks (PINNs) have shown promise in alleviating CoD in solving PDEs, vanilla PINNs fall short in learning discontinuous solutions due to their sampling nature, leading to poor safety performance of the resulting policies when values are discontinuous due to state or temporal logic constraints. In this study, we explore three potential solutions to this challenge: (1) a hybrid learning method that is guided by both supervisory equilibria and the HJI PDE, (2) a value-hardening method where a sequence of HJIs are solved with increasing Lipschitz constant on the constraint violation penalty, and (3) the epigraphical technique that lifts the value to a higher dimensional state space where it becomes continuous. Evaluations through 5D and 9D vehicle and 13D drone simulations reveal that the hybrid method outperforms others in terms of generalization and safety performance by taking advantage of both the supervisory equilibrium values and costates, and the low cost of PINN loss gradients.

Figures (16)

• Figure 1: (a) Value comparison among the learning methods for a simple 1D case. Red dots are the supervised data. (b) Evolution of the value function due to gradually hardening delta function. Delta functions are shown on top. Transparency reduces with hardening.
• Figure 2: (a) State trajectories of players projected to $(d_1, d_2)$. Solid gray box: collision area from the perspective of aggressive players; hollow boxes (magenta for Player 1 and blue for Player 2): collision areas from the perspectives of non-aggressive players. Red box: sampling domain for initial states. Color: Actual values of Player 1. (b) Uncontrolled intersection setup.
• Figure 3: (a), (g): Ground truth trajectories (projected to $d_1$-$d_2$) for $\mathcal{X}_{GT}$ and $\mathcal{X}_{XP}$, respectively. (b-f), (h-l): Trajectories generated using hybrid, value hardening, epigraphical, supervised, and vanilla PINN methods under $\mathcal{X}_{GT}$ and $\mathcal{X}_{XP}$, respectively. Color: Actual equilibrial values of Player 1 along the trajectories. Trajectories with inevitable collisions are removed for clearer comparison on safety performance. Red dots represent initial states with avoidable collisions.
• Figure 4: (a) Ground truth safe/unsafe initial states projected to $d_1$-$d_2$ frame, where black dots represent collision-free trajectories while orange dots depict trajectories with collision. (b) Value contours at initial time to classify safe/unsafe zones using HL. (c-h): Value contours along time using EL. Blue (red) regions represent unsafe (safe) states. (i) Comparison of mean and standard deviation of $|u-\hat{u}|$ from HL and EL across test trajectories sampled from $\mathcal{X}_{GT}$.
• Figure 5: Trajectories generated using neural networks with (a) relu and (b) sin activation functions and using $L^1$ for boundary norm (for tanh, refer to Fig. \ref{['fig:complete info']}); trajectories generated using (c) $L^1$- and (d) $L^{2}$-norms for the boundary values and using $\texttt{tanh}$ for activation. All trajectories are based on hybrid learning.

Theorems & Definitions (6)

• Lemma 1
• Lemma 2
• Theorem 1: HJ PDE with state constraints for general-sum differential games
• proof
• proof
• proof