Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach

Anton Plaksin, Vitaly Kalev

TL;DR

This work reframes Robust Reinforcement Learning as a zero-sum, finite-horizon differential game (a positional differential game) and proves that, under Isaacs's condition, a single shared Q-function can approximately satisfy both minimax and maximin Bellman equations. Leveraging this, the authors derive Isaacs DQN (IDQN) and Decomposed Isaacs DQN (DIDQN) as centralized, pure-policy extensions of Deep Q-Learning, showing improved robustness and performance over baselines in diverse differential-game environments. The key contributions are: (i) a theoretical result enabling a shared Q-function for differential games, (ii) practical IDQN and DIDQN algorithms with empirical superiority over RRL and MARL baselines, and (iii) a comprehensive evaluation framework and diverse testbed spanning classical differential games and MuJoCo-based tasks. The findings suggest centralized, Q-function-sharing approaches, grounded in differential-game theory, can provide robust and stable policies for real-world RL under disturbances, with potential impact on safety-critical and uncertain environments.

Abstract

Robust Reinforcement Learning (RRL) is a promising Reinforcement Learning (RL) paradigm aimed at training robust to uncertainty or disturbances models, making them more efficient for real-world applications. Following this paradigm, uncertainty or disturbances are interpreted as actions of a second adversarial agent, and thus, the problem is reduced to seeking the agents' policies robust to any opponent's actions. This paper is the first to propose considering the RRL problems within the positional differential game theory, which helps us to obtain theoretically justified intuition to develop a centralized Q-learning approach. Namely, we prove that under Isaacs's condition (sufficiently general for real-world dynamical systems), the same Q-function can be utilized as an approximate solution of both minimax and maximin Bellman equations. Based on these results, we present the Isaacs Deep Q-Network algorithms and demonstrate their superiority compared to other baseline RRL and Multi-Agent RL algorithms in various environments.

Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach

TL;DR

This work reframes Robust Reinforcement Learning as a zero-sum, finite-horizon differential game (a positional differential game) and proves that, under Isaacs's condition, a single shared Q-function can approximately satisfy both minimax and maximin Bellman equations. Leveraging this, the authors derive Isaacs DQN (IDQN) and Decomposed Isaacs DQN (DIDQN) as centralized, pure-policy extensions of Deep Q-Learning, showing improved robustness and performance over baselines in diverse differential-game environments. The key contributions are: (i) a theoretical result enabling a shared Q-function for differential games, (ii) practical IDQN and DIDQN algorithms with empirical superiority over RRL and MARL baselines, and (iii) a comprehensive evaluation framework and diverse testbed spanning classical differential games and MuJoCo-based tasks. The findings suggest centralized, Q-function-sharing approaches, grounded in differential-game theory, can provide robust and stable policies for real-world RL under disturbances, with potential impact on safety-critical and uncertain environments.

Abstract

Robust Reinforcement Learning (RRL) is a promising Reinforcement Learning (RL) paradigm aimed at training robust to uncertainty or disturbances models, making them more efficient for real-world applications. Following this paradigm, uncertainty or disturbances are interpreted as actions of a second adversarial agent, and thus, the problem is reduced to seeking the agents' policies robust to any opponent's actions. This paper is the first to propose considering the RRL problems within the positional differential game theory, which helps us to obtain theoretically justified intuition to develop a centralized Q-learning approach. Namely, we prove that under Isaacs's condition (sufficiently general for real-world dynamical systems), the same Q-function can be utilized as an approximate solution of both minimax and maximin Bellman equations. Based on these results, we present the Isaacs Deep Q-Network algorithms and demonstrate their superiority compared to other baseline RRL and Multi-Agent RL algorithms in various environments.
Paper Structure (35 sections, 1 theorem, 71 equations, 3 figures, 2 tables)

This paper contains 35 sections, 1 theorem, 71 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Positional Differential Games
  4. Shared Q-function for Approximate Bellman Equations
  5. Two-agent Deep Q-Networks
  6. Baseline Zero-Sum Algorithms
  7. NashDQN.
  8. Multi-agent DQN (MADQN).
  9. CounterDQN.
  10. Our Algorithms
  11. Isaacs's DQN (IDQN).
  12. Decomposed Isaacs's DQN (DIDQN)
  13. Experiments
  14. Algorithms.
  15. Environments.
  16. ...and 20 more sections

Key Result

Theorem 4.1

Let Isaacs's condition (saddle_point_condition) holds. Let the value function $V(\tau,w)$ be continuously differentiable at every $(\tau,w) \in [0,T] \times \mathbb R^n$. Then the following statements are valid: a) For every compact set $D \subset \mathbb R^n$ and $\varepsilon > 0$, there exists $\d for any $(t_i,x) \in \Delta \times D$, $i \neq m+1$, $u \in \mathcal{U}$, $v \in \mathcal{V}$, wher

Figures (3)

  • Figure 1: Visualization of the second agent's actions in the games based on the MuJoCo tasks
  • Figure 2: Evaluation scheme of trained policies based on using various RL algorithms with various parameters (see Evaluation scheme).
  • Figure 3: Experimental results of the considered 8 algorithms (see Algorithms) in 8 differential games (see Environments).

Theorems & Definitions (2)

  • Theorem 4.1
  • Remark 4.2