Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Historical Interaction-Enhanced Shapley Policy Gradient Algorithm for Multi-Agent Credit Assignment

Ao Ding, Licheng Sun, Yongjie Hou, Huaqing Zhang, Hongbin Ma

TL;DR

This work tackles credit assignment in cooperative multi-agent reinforcement learning by introducing HIS, a hybrid mechanism that blends a global baseline with Shapley-based incentives derived from historical interactions. The approach provides theoretical guarantees of efficiency and core stability while employing Approximate Marginal Contribution to estimate Shapley values in a sample-efficient manner, supplemented by Box-Cox normalization to stabilize learning. Empirically, HIS outperforms state-of-the-art baselines across three continuous-domain benchmarks (MPE, Bi-DexHands, MAMuJoCo), especially in strongly coupled tasks, and ablations confirm the importance of both the hybrid payoff structure and history-based credit signals. These results underscore the practical impact of combining explicit Shapley-based credit with stable global rewards for robust, scalable multi-agent collaboration.

Abstract

Multi-agent reinforcement learning (MARL) has demonstrated remarkable performance in multi-agent collaboration problems and has become a prominent topic in artificial intelligence research in recent years. However, traditional credit assignment schemes in MARL cannot reliably capture individual contributions in strongly coupled tasks while maintaining training stability, which leads to limited generalization capabilities and hinders algorithm performance. To address these challenges, we propose a Historical Interaction-Enhanced Shapley Policy Gradient Algorithm (HIS) for Multi-Agent Credit Assignment, which employs a hybrid credit assignment mechanism to balance base rewards with individual contribution incentives. By utilizing historical interaction data to calculate the Shapley value in a sample-efficient manner, HIS enhances the agent's ability to perceive its own contribution, while retaining the global reward to maintain training stability. Additionally, we provide theoretical guarantees for the hybrid credit assignment mechanism, ensuring that the assignment results it generates are both efficient and stable. We evaluate the proposed algorithm in three widely used continuous-action benchmark environments: Multi-Agent Particle Environment, Multi-Agent MuJoCo, and Bi-DexHands. Experimental results demonstrate that HIS outperforms state-of-the-art methods, particularly excelling in strongly coupled, complex collaborative tasks.

A Historical Interaction-Enhanced Shapley Policy Gradient Algorithm for Multi-Agent Credit Assignment

TL;DR

This work tackles credit assignment in cooperative multi-agent reinforcement learning by introducing HIS, a hybrid mechanism that blends a global baseline with Shapley-based incentives derived from historical interactions. The approach provides theoretical guarantees of efficiency and core stability while employing Approximate Marginal Contribution to estimate Shapley values in a sample-efficient manner, supplemented by Box-Cox normalization to stabilize learning. Empirically, HIS outperforms state-of-the-art baselines across three continuous-domain benchmarks (MPE, Bi-DexHands, MAMuJoCo), especially in strongly coupled tasks, and ablations confirm the importance of both the hybrid payoff structure and history-based credit signals. These results underscore the practical impact of combining explicit Shapley-based credit with stable global rewards for robust, scalable multi-agent collaboration.

Abstract

Multi-agent reinforcement learning (MARL) has demonstrated remarkable performance in multi-agent collaboration problems and has become a prominent topic in artificial intelligence research in recent years. However, traditional credit assignment schemes in MARL cannot reliably capture individual contributions in strongly coupled tasks while maintaining training stability, which leads to limited generalization capabilities and hinders algorithm performance. To address these challenges, we propose a Historical Interaction-Enhanced Shapley Policy Gradient Algorithm (HIS) for Multi-Agent Credit Assignment, which employs a hybrid credit assignment mechanism to balance base rewards with individual contribution incentives. By utilizing historical interaction data to calculate the Shapley value in a sample-efficient manner, HIS enhances the agent's ability to perceive its own contribution, while retaining the global reward to maintain training stability. Additionally, we provide theoretical guarantees for the hybrid credit assignment mechanism, ensuring that the assignment results it generates are both efficient and stable. We evaluate the proposed algorithm in three widely used continuous-action benchmark environments: Multi-Agent Particle Environment, Multi-Agent MuJoCo, and Bi-DexHands. Experimental results demonstrate that HIS outperforms state-of-the-art methods, particularly excelling in strongly coupled, complex collaborative tasks.

Paper Structure

This paper contains 36 sections, 6 theorems, 34 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Cooperative Multi-Agent Reinforcement Learning
  5. Convex Game
  6. Shapley Value
  7. BOX-COX Transformation
  8. Method
  9. Hybrid Credit Assignment Mechanism
  10. Historical Interaction Information-Based Shapley Q-value Policy Gradient
  11. Historical Interaction-Enhanced Shapley Policy Gradient
  12. Practical Algorithm
  13. Experiments
  14. Experimental Setup
  15. Experimental Results
  16. ...and 21 more sections

Key Result

Theorem 1

Given a convex game $G=(\mathcal{N},v)$, the Shapley value must belong to the Core, and the assignment scheme is stable.

Figures (8)

  • Figure 1: Schematic diagram of the motivation and overall process of our proposed HIS method.
  • Figure 2: The performance of algorithms on different tasks in multiple benchmark environments.
  • Figure 3: Ablation study of different HIS components on Bi-DexHands and MAMuJoCo tasks.
  • Figure 4: Performance of algorithms in different tasks in Bi-DexHands.
  • Figure 5: Performance of algorithms in different tasks in MAMuJoCo.
  • ...and 3 more figures

Theorems & Definitions (14)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Definition 5
  • Definition 6
  • Definition 7
  • ...and 4 more