Differentially Private Reinforcement Learning with Self-Play

TL;DR

The definitions of Joint DP (JDP) and Local DP (LDP) are extended to two-player zero-sum episodic Markov Games and a provably efficient algorithm is designed based on optimistic Nash value iteration and privatization of Bernstein-type bonuses to understand trajectory-wise privacy protection in multi-agent RL.

Abstract

We study the problem of multi-agent reinforcement learning (multi-agent RL) with differential privacy (DP) constraints. This is well-motivated by various real-world applications involving sensitive data, where it is critical to protect users' private information. We first extend the definitions of Joint DP (JDP) and Local DP (LDP) to two-player zero-sum episodic Markov Games, where both definitions ensure trajectory-wise privacy protection. Then we design a provably efficient algorithm based on optimistic Nash value iteration and privatization of Bernstein-type bonuses. The algorithm is able to satisfy JDP and LDP requirements when instantiated with appropriate privacy mechanisms. Furthermore, for both notions of DP, our regret bound generalizes the best known result under the single-agent RL case, while our regret could also reduce to the best known result for multi-agent RL without privacy constraints. To the best of our knowledge, these are the first line of results towards understanding trajectory-wise privacy protection in multi-agent RL.

Theorems & Definitions (39)

• Definition 2.1: Differential Privacy (DP)
• Definition 2.2: Joint Differential Privacy (JDP)
• Definition 2.3: Local Differential Privacy (LDP)
• Remark 2.4
• Remark 2.5
• Remark 3.2
• Theorem 4.1
• Theorem 4.2
• Lemma 5.1
• Theorem 5.2: Results under JDP