Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning

TL;DR

This work addresses the challenge of achieving a fully expressive yet tractable value function decomposition that satisfies the individual-global max (IGM) property in cooperative multi-agent reinforcement learning. It introduces a simple, general formulation of the IGM-complete function class and derives QFIX, a lightweight family of fixing networks that expand the representational capacity of non-IGM-complete baselines by applying a single, positive-weighted transformation to the fixee’s advantages. Variants of QFIX (including QFIX-sum, QFIX-mono, and QFIX-lin) and additive QFIX (Q+FIX) are developed, along with stateful adaptations; they are shown to achieve IGM-completeness and, in experiments on SMACv2 and Overcooked, to outperform or closely match state-of-the-art QPLEX while using smaller, simpler mixing architectures. Empirical results demonstrate improved stability and performance over VDN and QMIX, with competitive results to QPLEX in many tasks, and ablations indicating gains arise from the fixing mechanism rather than added parameters. Overall, the QFIX framework offers a principled, scalable path to richer IGM-complete decompositions, enabling better coordination in decentralized MARL settings.

Abstract

Value function decomposition methods for cooperative multi-agent reinforcement learning compose joint values from individual per-agent utilities, and train them using a joint objective. To ensure that the action selection process between individual utilities and joint values remains consistent, it is imperative for the composition to satisfy the individual-global max (IGM) property. Although satisfying IGM itself is straightforward, most existing methods (e.g., VDN, QMIX) have limited representation capabilities and are unable to represent the full class of IGM values, and the one exception that has no such limitation (QPLEX) is unnecessarily complex. In this work, we present a simple formulation of the full class of IGM values that naturally leads to the derivation of QFIX, a novel family of value function decomposition models that expand the representation capabilities of prior models by means of a thin "fixing" layer. We derive multiple variants of QFIX, and implement three variants in two well-known multi-agent frameworks. We perform an empirical evaluation on multiple SMACv2 and Overcooked environments, which confirms that QFIX (i) succeeds in enhancing the performance of prior methods, (ii) learns more stably and performs better than its main competitor QPLEX, and (iii) achieves this while employing the simplest and smallest mixing models.

Figures (8)

• Figure 1: Diagrams for QFIX (left) and Q+FIX (right).
• Figure 2: SMACv2 results, bootstrapped $95\%$ CI. Aggregate returns are normalized per-task via $\tilde{G}_i \doteq \frac{G_i - \min_k G_k}{\max_k G_k - \min_k G_k}$, where $\{ G_i \}_i$ is the total set of returns logged by all models in a given task.
• Figure 3: Overcooked return mean, bootstrapped $95\%$ CI ($20$ seeds).
• Figure 4: Specialized diagrams for Q+FIX-sum, Q+FIX-mono, and Q+FIX-lin.
• Figure 5: SMACv2 winrate results, bootstrapped $95\%$ CI.

Theorems & Definitions (18)

• definition 1: Individual-Global Max
• definition 2: IGM Function Class
• proposition 1: Advantage Constraints
• definition 3: Stateful-IGM
• proposition 2: Simplified Advantage Constraints
• proposition 3
• proposition 4
• proposition 5
• proof
• proof : Proof by mutual inclusion