Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning

Bang Giang Le, Viet Cuong Ta

TL;DR

This work tackles credit assignment in cooperative multi-agent reinforcement learning by introducing a dependence-graph framework that uses local rewards while maintaining cooperation. It develops a Dependence Graph Policy Gradient that only accounts for causal interaction paths, and provides a practical, model-based method to estimate the dependence graph via reverse world models and a latent encoder, paired with a GAElike gradient aggregation across meeting timesteps. Empirical results on Level-Based Foraging and SMAClite show the approach can outperform purely local or global reward baselines and remain robust to graph estimation errors, highlighting the method’s flexibility and scalability. The combination of theoretical grounding and practical graph-estimation techniques offers a principled, adaptable path toward improved coordination and learning efficiency in complex MARL environments.

Abstract

To promote cooperation in Multi-Agent Reinforcement Learning, the reward signals of all agents can be aggregated together, forming global rewards that are commonly known as the fully cooperative setting. However, global rewards are usually noisy because they contain the contributions of all agents, which have to be resolved in the credit assignment process. On the other hand, using local reward benefits from faster learning due to the separation of agents' contributions, but can be suboptimal as agents myopically optimize their own reward while disregarding the global optimality. In this work, we propose a method that combines the merits of both approaches. By using a graph of interaction between agents, our method discerns the individual agent contribution in a more fine-grained manner than a global reward, while alleviating the cooperation problem with agents' local reward. We also introduce a practical approach for approximating such a graph. Our experiments demonstrate the flexibility of the approach, enabling improvements over the traditional local and global reward settings.

Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning

TL;DR

This work tackles credit assignment in cooperative multi-agent reinforcement learning by introducing a dependence-graph framework that uses local rewards while maintaining cooperation. It develops a Dependence Graph Policy Gradient that only accounts for causal interaction paths, and provides a practical, model-based method to estimate the dependence graph via reverse world models and a latent encoder, paired with a GAElike gradient aggregation across meeting timesteps. Empirical results on Level-Based Foraging and SMAClite show the approach can outperform purely local or global reward baselines and remain robust to graph estimation errors, highlighting the method’s flexibility and scalability. The combination of theoretical grounding and practical graph-estimation techniques offers a principled, adaptable path toward improved coordination and learning efficiency in complex MARL environments.

Abstract

To promote cooperation in Multi-Agent Reinforcement Learning, the reward signals of all agents can be aggregated together, forming global rewards that are commonly known as the fully cooperative setting. However, global rewards are usually noisy because they contain the contributions of all agents, which have to be resolved in the credit assignment process. On the other hand, using local reward benefits from faster learning due to the separation of agents' contributions, but can be suboptimal as agents myopically optimize their own reward while disregarding the global optimality. In this work, we propose a method that combines the merits of both approaches. By using a graph of interaction between agents, our method discerns the individual agent contribution in a more fine-grained manner than a global reward, while alleviating the cooperation problem with agents' local reward. We also introduce a practical approach for approximating such a graph. Our experiments demonstrate the flexibility of the approach, enabling improvements over the traditional local and global reward settings.
Paper Structure (27 sections, 3 theorems, 44 equations, 9 figures, 1 table, 2 algorithms)

This paper contains 27 sections, 3 theorems, 44 equations, 9 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setting and Notation
  4. Motivation: Scalarize or not scalarize
  5. Method
  6. Dependence Graph Policy Gradient
  7. Dependence Graph Estimation as a World Model learning problem
  8. Approximating Dependence Graph via reverse world models
  9. Practical Implementation
  10. Experiments
  11. Results
  12. Ablation study on different graph structures
  13. Conclusions
  14. Impact Statement
  15. Appendix
  16. ...and 12 more sections

Key Result

Proposition 5.1

Fix a joint policy $\boldsymbol{\pi}$. Let $i, j \in \mathcal{N}$. The policy gradient $\nabla_{\pi_j}\mathcal{J}^i(\boldsymbol{\pi})$ is given by where $t'$ is the first timestep where there is a path from $(\mathbf s_0, j)$ to $(\mathbf s_{t'}, i)$ in a trajectory $\tau$. A path is defined as in definition def:path.

Figures (9)

  • Figure 1: Reward dilemma in cooperative MARL; global reward enhances cooperation but introduces credit assignment problem, while local reward can induce suboptimal policies in the environments that require cooperation. Our method enables faster training, as in local rewards, while avoiding the miscoordination pitfall.
  • Figure 2: An example of an MDP with decomposed state structure. Agent $k$ (top right) cannot influence agent $q$ at timestep $t+1$; any effect can only occur from timestep $t+2$ onward (bottom). Since agent $q$ lies in agent $k$'s blind spot, it can be excluded from the gradient computation at $t+1$.
  • Figure 3: Dependence graph approximation via reverse world models.
  • Figure 4: Results on the LBF benchmark on 6 selected scenarios. We follow the evaluation protocol recommended in agarwal2021deep. Methods with dependence graphs consistently outperform other baselines from both local and global reward settings.
  • Figure 5: Result on the SMAClite benchmark on 6 scenarios. Overall, the performances of all methods are relatively comparable.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Definition 3.1: State Dependence Graph induced by a Networked Multi-Agent MDP
  • Definition 3.2: Proper Dependence Graph
  • Definition 3.3: Path in the Dependence Graph
  • Proposition 5.1
  • Lemma 5.2: Gradient Error under an Approximated Dependence Graph
  • proof : Proof of Proposition \ref{['prop:global']}
  • proof : Proof of Lemma \ref{['lem:graph-approx']}
  • proof : Derivation of the inequality \ref{['eq:mutual_info_bound']}
  • Proposition 1.1