Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Reinforcement Learning with Submodular Reward

Wenjing Chen, Chengyuan Qian, Shuo Xing, Yi Zhou, Victoria Crawford

TL;DR

This paper provides the first formal framework for cooperative multi-agent reinforcement learning (MARL) where the joint reward exhibits submodularity, which is a natural property capturing diminishing marginal returns when adding agents to a team.

Abstract

In this paper, we study cooperative multi-agent reinforcement learning (MARL) where the joint reward exhibits submodularity, which is a natural property capturing diminishing marginal returns when adding agents to a team. Unlike standard MARL with additive rewards, submodular rewards model realistic scenarios where agent contributions overlap (e.g., multi-drone surveillance, collaborative exploration). We provide the first formal framework for this setting and develop algorithms with provable guarantees on sample efficiency and regret bound. For known dynamics, our greedy policy optimization achieves a $1/2$-approximation with polynomial complexity in the number of agents $K$, overcoming the exponential curse of dimensionality inherent in joint policy optimization. For unknown dynamics, we propose a UCB-based learning algorithm achieving a $1/2$-regret of $O(H^2KS\sqrt{AT})$ over $T$ episodes.

Multi-Agent Reinforcement Learning with Submodular Reward

TL;DR

This paper provides the first formal framework for cooperative multi-agent reinforcement learning (MARL) where the joint reward exhibits submodularity, which is a natural property capturing diminishing marginal returns when adding agents to a team.

Abstract

In this paper, we study cooperative multi-agent reinforcement learning (MARL) where the joint reward exhibits submodularity, which is a natural property capturing diminishing marginal returns when adding agents to a team. Unlike standard MARL with additive rewards, submodular rewards model realistic scenarios where agent contributions overlap (e.g., multi-drone surveillance, collaborative exploration). We provide the first formal framework for this setting and develop algorithms with provable guarantees on sample efficiency and regret bound. For known dynamics, our greedy policy optimization achieves a -approximation with polynomial complexity in the number of agents , overcoming the exponential curse of dimensionality inherent in joint policy optimization. For unknown dynamics, we propose a UCB-based learning algorithm achieving a -regret of over episodes.
Paper Structure (44 sections, 15 theorems, 149 equations, 1 figure, 2 algorithms)

This paper contains 44 sections, 15 theorems, 149 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Agent Reinforcement Learning
  4. Submodular Maximization under Matroid Constraint
  5. Preliminaries
  6. Multi-Agent MDP with Submodular Reward
  7. Episode Dynamics
  8. Submodular Rewards
  9. Main Results
  10. Computational Challenges and Value Function Decomposition
  11. Computational Challenges
  12. Joint Policies and Reward Decomposition
  13. Marginal Value and $Q$-Functions
  14. Greedy Policy Optimization for known $P$
  15. UCB-Based Greedy Value Iteration for Unknown Transition Dynamics
  16. ...and 29 more sections

Key Result

Lemma 1

For any policy $\pi_i$, $h\in[H]$, $s\in\mathcal{S}$, and $a\in\mathcal{A}$, Moreover, let $\widetilde{\pi}_i=\arg\max_{\pi_i}\mathbb{E}_{\pi_i}[\sum_{t=1}^H \Delta r_i(\mathbf{s}_t,\mathbf{a}_t)|s_1^i=\bar{s}_1^i,\pi_{[i-1]}]$. Then

Figures (1)

  • Figure 1: Demonstration of multi-agent collaboration.

Theorems & Definitions (29)

  • Definition 1: Multi-Agent Markov Decision Processes with Submodular Reward (MAMDP-SR)
  • Definition 2
  • Lemma 1: Bellman Equation for Marginal Gains
  • Theorem 1
  • Definition 3: $\alpha$-regret
  • Theorem 2: Regret bound for UCB-GVI
  • Lemma 2: Bellman Optimality bellman1957markoviansutton1998reinforcement
  • Lemma 3: Equivalence to Partition Matroid Submodular Maximization
  • proof
  • proof : Proof sketch
  • ...and 19 more