MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Trees

TL;DR

MIXRTs address interpretability gaps in multi-agent reinforcement learning by introducing Recurrent Tree Cells (RTCs) and a linear mixing architecture that factorizes the joint action-value $Q_{\mathrm{tot}}$ into agent-specific $Q_i$ with positive weights $W_i$. The RTCs integrate history through recurrence and are ensemble-merged to maintain expressivity while preserving interpretability; the mixing trees guarantee additivity and monotonicity, ensuring IGM-consistent decentralized actions. Experiments on Spread and StarCraft II SMAC show MIXRTs achieve competitive performance relative to strong baselines while providing explicit, root-to-leaf explanations and feature-importance insights. Overall, the work demonstrates that interpretable, high-performing MARL systems are attainable and offers practical tools for understanding agent credits and cooperative strategies.

Abstract

While achieving tremendous success in various fields, existing multi-agent reinforcement learning (MARL) with a black-box neural network makes decisions in an opaque manner that hinders humans from understanding the learned knowledge and how input observations influence decisions. In contrast, existing interpretable approaches usually suffer from weak expressivity and low performance. To bridge this gap, we propose MIXing Recurrent soft decision Trees (MIXRTs), a novel interpretable architecture that can represent explicit decision processes via the root-to-leaf path and reflect each agent's contribution to the team. Specifically, we construct a novel soft decision tree using a recurrent structure and demonstrate which features influence the decision-making process. Then, based on the value decomposition framework, we linearly assign credit to each agent by explicitly mixing individual action values to estimate the joint action value using only local observations, providing new insights into interpreting the cooperation mechanism. Theoretical analysis confirms that MIXRTs guarantee additivity and monotonicity in the factorization of joint action values. Evaluations on complex tasks like Spread and StarCraft II demonstrate that MIXRTs compete with existing methods while providing clear explanations, paving the way for interpretable and high-performing MARL systems.

Figures (11)

• Figure 1: Examples of SDT and RTC. (a) A two-level SDT. (b) Illustration of the process of a two-level RTC that receives the current individual observation $o^{t}_{i}$ and the previous hidden state $h^{t-1}_{i}$ as input at each timestamp.
• Figure 2: MIXRTs architecture. (a) Diagram of the structure of the mixing tree with depth $2$. (b) In the overall MIXRTs architecture, we finally obtain the joint $Q_{tot}$ value via a linear combination of the individual action-value functions. (c) Individual RTCs for each agent.
• Figure 3: Median test occupied landmarks % across different numbers of agents in the Spread environment.
• Figure 4: Median test win rates % for easy, hard and super-hard scenarios of SMAC.
• Figure 5: Comparison of I-SDTs, I-CDTs, I-RTCs, and MIXRTs with different depths.

Theorems & Definitions (1)

• proof