Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning

Qinyu Xu, Yuanyang Zhu, Xuefei Wu, Chunlin Chen

TL;DR

QCoFr introduces a principled, interpretable approach to MARL that models high-order agent interactions with a CFN-based mixer, achieving linear complexity in the number of agents. By coupling continued fraction mixers with a variational information bottleneck for assistive information, it enables accurate credit assignment and robust cooperation. The method delivers strong, state-of-the-art performance on LBF, SMAC, and SMACv2 benchmarks while providing interpretable attributions to individual agents and coalitions. This work advances scalable, transparent MARL by linking depth of interaction modeling to order, via Padé-like rational approximants, and demonstrates practical impact in complex coordination tasks.

Abstract

The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity $\mathcal{O}\left({n}\right)$ in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.

High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning

TL;DR

QCoFr introduces a principled, interpretable approach to MARL that models high-order agent interactions with a CFN-based mixer, achieving linear complexity in the number of agents. By coupling continued fraction mixers with a variational information bottleneck for assistive information, it enables accurate credit assignment and robust cooperation. The method delivers strong, state-of-the-art performance on LBF, SMAC, and SMACv2 benchmarks while providing interpretable attributions to individual agents and coalitions. This work advances scalable, transparent MARL by linking depth of interaction modeling to order, via Padé-like rational approximants, and demonstrates practical impact in complex coordination tasks.

Abstract

The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.
Paper Structure (25 sections, 7 theorems, 63 equations, 14 figures, 4 tables, 1 algorithm)

This paper contains 25 sections, 7 theorems, 63 equations, 14 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. High-order Interactions Modeling for Decomposition
  4. Methods
  5. Experiments
  6. Performance Comparison
  7. Ablation Studies
  8. Interpretability
  9. Conclusion
  10. Acknowledgements
  11. Related Work
  12. Proof
  13. Objective Functions for Variational Information Bottlenecks
  14. Correspondence between Continued Fraction Depth and the Order of Agent Interactions
  15. Experimental Details
  16. ...and 10 more sections

Key Result

Theorem 1

Let the representation $m_i$ be reparameterized as a random variable drawn from a multivariate Gaussian distribution $m_i \sim \mathcal{N}(f_m(h_i; \phi_m), \boldsymbol{I})$, where $f_m$ is an encoder parameterized by $\phi_m$, $h_i$ denotes the hidden state of agent $i$, and $\boldsymbol{I}$ is the where $q(u^*_i \mid m_i)$ is a variational distribution approximating the true posterior $p(u^*_i \

Figures (14)

  • Figure 1: A diagram of CFN composed of different depth ladders that take individual values $\boldsymbol{Q}$ as inputs and output temporal values $\hat{Q}$ with $d$-order interactions.
  • Figure 2: The overall architecture of QCoFr. A CFN-based mixing network models high-order interactions among agents by expressing $Q_{\text{tot}}$ as a linear combination of $l$ ladders over individual values $\boldsymbol{Q}$. The VIB module encodes the hidden state $\boldsymbol{h}$ into assistive information $\boldsymbol{m}$, which is used to deduce the credits of each agent of coalitions together with the global state $\boldsymbol{s}$.
  • Figure 3: Performance results on LBF benchmark.
  • Figure 4: Performance results on SMAC benchmark.
  • Figure 5: Performance results on SMACv2 benchmark.
  • ...and 9 more figures

Theorems & Definitions (14)

  • Definition 1: Padé Approximant baker1961padelorentzen2010pade
  • Theorem 1: Lower Bound for $I(\boldsymbol{m}, \boldsymbol{u^*}; \boldsymbol{\phi})$
  • Theorem 2: Upper Bound for $I(\boldsymbol{m}, \boldsymbol{h}; \boldsymbol{\phi})$
  • Theorem 1: Lower Bound for $I(\boldsymbol{m}, \boldsymbol{u^*}; \boldsymbol{\phi})$
  • proof
  • Theorem 2: Upper Bound for $I(\boldsymbol{m}, \boldsymbol{h}; \boldsymbol{\phi})$
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • ...and 4 more