Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Partial Action Replacement in Offline MARL

Yue Jin, Giovanni Montana

Abstract

Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.

Learning Partial Action Replacement in Offline MARL

Abstract

Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.

Paper Structure

This paper contains 15 sections, 1 theorem, 11 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. PAR Policy
  6. The PLCQL Algorithm
  7. Theoretical Value-Error Bound of PLCQL
  8. Experimental Settings and Results
  9. Experimental Setup
  10. Baselines
  11. Main Results
  12. Ablation Study
  13. Computational Cost
  14. Discussion and Conclusions
  15. Generative AI disclosure.

Key Result

theorem 1

Let $\hat{Q}$ be the function learned by PLCQL and define the average single-agent policy deviation as $\overline{\mathrm{TV}}(\pi, \mu) = \frac{1}{n} \sum_{i=1}^n \mathrm{TV}(\pi_i, \mu_i)$. The value estimation error is bounded by:

Figures (1)

  • Figure 1: Illustration of the PLCQL algorithm.

Theorems & Definitions (2)

  • theorem 1: PLCQL Value-Error Bound
  • proof : Proof Sketch