Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning

Tianmeng Hu, Yongzheng Cui, Rui Tang, Biao Luo, Ke Li

TL;DR

This work addresses IGM consistency in multi-agent value function factorization by adopting a dynamical-systems lens to non-monotonic value decomposition under CTDE. It introduces a continuous-time gradient-flow analysis showing that, with approximately greedy exploration, only $IGM$-consistent fixed points are stable while $IGM$-inconsistent ones are unstable saddles, and validates this with matrix-game and MARL benchmarks. The methodology combines a non-monotonic mixing function, SARSA-style TD($\lambda$) targets, and Random Network Distillation-based intrinsic rewards to empirically realize the theoretical insights, achieving superior performance over monotonic baselines on SMAC and GRF. These results suggest that relaxing monotonic constraints, when paired with appropriate exploration and learning dynamics, can yield more expressive and effective value-based MARL algorithms.

Abstract

Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure individual-global-max (IGM) consistency, existing methods either enforce monotonicity constraints, which limit expressive power, or adopt softer surrogates at the cost of algorithmic complexity. In this work, we present a dynamical systems analysis of non-monotonic value decomposition, modeling learning dynamics as continuous-time gradient flow. We prove that, under approximately greedy exploration, all zero-loss equilibria violating IGM consistency are unstable saddle points, while only IGM-consistent solutions are stable attractors of the learning dynamics. Extensive experiments on both synthetic matrix games and challenging MARL benchmarks demonstrate that unconstrained, non-monotonic factorization reliably recovers IGM-optimal solutions and consistently outperforms monotonic baselines. Additionally, we investigate the influence of temporal-difference targets and exploration strategies, providing actionable insights for the design of future value-based MARL algorithms.

Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning

TL;DR

This work addresses IGM consistency in multi-agent value function factorization by adopting a dynamical-systems lens to non-monotonic value decomposition under CTDE. It introduces a continuous-time gradient-flow analysis showing that, with approximately greedy exploration, only -consistent fixed points are stable while -inconsistent ones are unstable saddles, and validates this with matrix-game and MARL benchmarks. The methodology combines a non-monotonic mixing function, SARSA-style TD() targets, and Random Network Distillation-based intrinsic rewards to empirically realize the theoretical insights, achieving superior performance over monotonic baselines on SMAC and GRF. These results suggest that relaxing monotonic constraints, when paired with appropriate exploration and learning dynamics, can yield more expressive and effective value-based MARL algorithms.

Abstract

Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure individual-global-max (IGM) consistency, existing methods either enforce monotonicity constraints, which limit expressive power, or adopt softer surrogates at the cost of algorithmic complexity. In this work, we present a dynamical systems analysis of non-monotonic value decomposition, modeling learning dynamics as continuous-time gradient flow. We prove that, under approximately greedy exploration, all zero-loss equilibria violating IGM consistency are unstable saddle points, while only IGM-consistent solutions are stable attractors of the learning dynamics. Extensive experiments on both synthetic matrix games and challenging MARL benchmarks demonstrate that unconstrained, non-monotonic factorization reliably recovers IGM-optimal solutions and consistently outperforms monotonic baselines. Additionally, we investigate the influence of temporal-difference targets and exploration strategies, providing actionable insights for the design of future value-based MARL algorithms.

Paper Structure

This paper contains 35 sections, 7 theorems, 60 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Dec-POMDPs
  4. Value Function Factorization
  5. IGM Consistency
  6. Analysis
  7. Fixed Uniform Policy
  8. Approximately Greedy Policy
  9. Methodology
  10. Non-Monotonic Mixing Function
  11. SARSA-Style Updating with TD($\lambda$)
  12. Intrinsic Reward Driven Exploration-Exploitation
  13. Experiments
  14. Matrix Game
  15. SMAC and GRF
  16. ...and 20 more sections

Key Result

Theorem 1

Under the fixed uniform policy, there exists a set of zero-loss points $\mathcal{M}_0 = \{\mathbf{q} \mid \mathcal{L}_0(\mathbf{q}) = 0\}$ containing infinitely many elements, and this set includes points that do not satisfy IGM consistency.

Figures (7)

  • Figure 1: Comparisons of test win rate on SMAC maps: 3s_vs_5z, corridor, and 3s5z_vs_3s6z. The results are averaged over five independent runs, with the 25%–75% interquartile range shown as a shaded region.
  • Figure 2: Comparisons of test win rate on GRF tasks: academy_3_vs_1_keeper, academy_counterattack_easy, and academy_counterattack_hard. The results are averaged over five independent runs.
  • Figure 3: Architecture of our method.
  • Figure 4: Architecture of RND.
  • Figure 5: Comparisons of test win rate on SMAC maps: 3s_vs_5z, and 3s5z_vs_3s6z. The results are averaged over five independent runs, with the 25%–75% interquartile range shown as a shaded region.
  • ...and 2 more figures

Theorems & Definitions (14)

  • Definition 1: Single-State Game
  • Theorem 1: Zero-Loss Points Without IGM Consistency
  • proof : Proof
  • Lemma 1
  • Theorem 2: Stability of IGM-Consistent Fixed Points
  • proof : Proof
  • Theorem 3: Instability of IGM-Inconsistent Fixed Points
  • proof : Proof
  • Lemma 2
  • proof
  • ...and 4 more