Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

Shuai Han, Mehdi Dastani, Shihan Wang

TL;DR

This work tackles sparse-reward multi-agent reinforcement learning by introducing Influence Scope of Agents (ISA), a framework that derives each agent's influence scope from mutual information between actions and state changes to enable precise credit assignment and targeted exploration. ISA partitions global goals into common and special segments based on agent influence, then trains goal-conditioned policies with intrinsic rewards aligned to these segments, while employing influence-scoped exploration bonuses to efficiently discover successful states. Across SMAC and MPE benchmarks, ISA outperforms state-of-the-art baselines and demonstrates ablations confirming the value of influence-scope-based credit assignment and exploration. The approach offers improved sample efficiency, interpretable credit attribution, and a principled way to manage high-dimensional joint action spaces in MARL, with potential for hierarchical extensions in future work.

Abstract

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods struggle with precise credit assignment among agents and effective exploration. In this paper, we introduce a novel method to deal with both credit assignment and exploration problems in reward-sparse domains. Accordingly, we propose an algorithm that calculates the Influence Scope of Agents (ISA) on states by taking specific value of the dimensions/attributes of states that can be influenced by individual agents. The mutual dependence between agents' actions and state attributes are then used to calculate the credit assignment and to delimit the exploration space for each individual agent. We then evaluate ISA in a variety of sparse-reward multi-agent scenarios. The results show that our method significantly outperforms the state-of-art baselines.

Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

TL;DR

This work tackles sparse-reward multi-agent reinforcement learning by introducing Influence Scope of Agents (ISA), a framework that derives each agent's influence scope from mutual information between actions and state changes to enable precise credit assignment and targeted exploration. ISA partitions global goals into common and special segments based on agent influence, then trains goal-conditioned policies with intrinsic rewards aligned to these segments, while employing influence-scoped exploration bonuses to efficiently discover successful states. Across SMAC and MPE benchmarks, ISA outperforms state-of-the-art baselines and demonstrates ablations confirming the value of influence-scope-based credit assignment and exploration. The approach offers improved sample efficiency, interpretable credit attribution, and a principled way to manage high-dimensional joint action spaces in MARL, with potential for hierarchical extensions in future work.

Abstract

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods struggle with precise credit assignment among agents and effective exploration. In this paper, we introduce a novel method to deal with both credit assignment and exploration problems in reward-sparse domains. Accordingly, we propose an algorithm that calculates the Influence Scope of Agents (ISA) on states by taking specific value of the dimensions/attributes of states that can be influenced by individual agents. The mutual dependence between agents' actions and state attributes are then used to calculate the credit assignment and to delimit the exploration space for each individual agent. We then evaluate ISA in a variety of sparse-reward multi-agent scenarios. The results show that our method significantly outperforms the state-of-art baselines.
Paper Structure (29 sections, 10 equations, 4 figures, 1 algorithm)

This paper contains 29 sections, 10 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Credit assignment in sparse-reward domains.
  4. Information-theoretic exploration or coordination.
  5. Preliminaries
  6. Dec-POMDP.
  7. Mutual information.
  8. Additional notations.
  9. Core Concepts
  10. Remark 1 (Fine-tunable influence scope of action):
  11. Remark 2 (Credit assignment based on influence):
  12. Remark 3: (Non-conflicting individual goals):
  13. Remark 4: (Trainable environments):
  14. Algorithm
  15. Step 1. Obtain influence scope.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Learning curves on SMAC (with only $+1/-1$ reward) and MPE.
  • Figure 2: An illustration on decomposing individual goals from a global goal.
  • Figure 3: Heat maps of mutual information values.
  • Figure 4: Ablations and interpretability for ISA

Theorems & Definitions (4)

  • Definition 1: Influence scope of action
  • Definition 2: Influence scope of agent
  • Definition 3: Global goal and individual goal
  • Definition 4: Common segment and special segment