Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learning

TL;DR

This work introduces information-directed sampling (IDS) for multi-agent reinforcement learning (MARL), addressing sample efficiency in episodic Markov games. It develops a suite of IDS-based algorithms—MAIDS, Reg-MAIDS, and Compressed-MAIDS—that balance exploitation and information gain about learning targets, achieving Bayesian regret on the order of $\tilde{O}(\sqrt{K})$ in two-player zero-sum MGs and extending to general-sum MGs. A key contribution is the use of learning targets beyond the full environment, notably compressed environments via rate-distortion-inspired constructions, which yield tighter regret bounds and computational savings. The methods rely on posterior sampling, mean-environment reductions, and an asymmetric learning scheme to cope with non-stationarity from other agents, offering practical, scalable approaches for NE/CCE learning in MARL with provable guarantees.

Abstract

This work designs and analyzes a novel set of algorithms for multi-agent reinforcement learning (MARL) based on the principle of information-directed sampling (IDS). These algorithms draw inspiration from foundational concepts in information theory, and are proven to be sample efficient in MARL settings such as two-player zero-sum Markov games (MGs) and multi-player general-sum MGs. For episodic two-player zero-sum MGs, we present three sample-efficient algorithms for learning Nash equilibrium. The basic algorithm, referred to as MAIDS, employs an asymmetric learning structure where the max-player first solves a minimax optimization problem based on the joint information ratio of the joint policy, and the min-player then minimizes the marginal information ratio with the max-player's policy fixed. Theoretical analyses show that it achieves a Bayesian regret of tilde{O}(sqrt{K}) for K episodes. To reduce the computational load of MAIDS, we develop an improved algorithm called Reg-MAIDS, which has the same Bayesian regret bound while enjoying less computational complexity. Moreover, by leveraging the flexibility of IDS principle in choosing the learning target, we propose two methods for constructing compressed environments based on rate-distortion theory, upon which we develop an algorithm Compressed-MAIDS wherein the learning target is a compressed environment. Finally, we extend Reg-MAIDS to multi-player general-sum MGs and prove that it can learn either the Nash equilibrium or coarse correlated equilibrium in a sample efficient manner.

Figures (1)

• Figure 1: Illustration of the MG in Example \ref{['example:compress']}

Theorems & Definitions (29)

• Theorem 1
• proof : Proof of Theorem \ref{['thm:maids']}
• Remark 1
• Remark 2
• Theorem 2
• proof : Proof of Theorem 2
• Lemma 1
• proof : Proof of Lemma \ref{['lemma:eq']}
• Example 1
• Remark 3